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1. INTRODUCTION 


The Total Learning Architecture (TLA) program sponsored by the ADL Initiative seeks to develop a set of 
policy and standards defining the process for developing a learning ecology, where multiple services and 
learning opportunities (of various modalities and points of delivery) can be managed in an integrated 
environment. The TLA serves to: 


1) shift from a course-centric focus to a learner-centric focus, where competence is managed more 
efficiently (for savings in cost and schedule) and effectively for the learner’s professional 
development path; 


2) provide the opportunities for data interoperability between implementations to aid in enterprise- 
wide decision support analysis and in reuse of competency and content between agencies, and; 


3) capture all experiences, including on-the-job training (OJT), use of performance support and job 
aids, simulation, distributed learning, and traditional classroom learning to provide a more rounded 
portrait of personnel capability and aptitude to aid in the team building and detailing process. 


The TLA employs a Service Oriented Architecture (SOA) approach to developing a flexible, modular 
system to optimize the time, cost, and effectiveness of learning delivery and analysis. It is composed of 
interchangeable software services and databases integrated to create an efficient learning environment for 
the user. The layered architecture of the TLA results in a separation of concerns amongst the different TLA 
components. Figure 1 shows the different layers within the TLA. These applications utilize common 
services and shared data to deliver instructional activities and provide overall management of the 
experience. TLA services are built around a set of defined set of specifications. 
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Figure 1. TLA Service Layers — The TLA’s service layers centralize external access to data and functions, hides the 
complexity of implementation across physical components, and allows for versioning of the services across the TLA 
ecosystem. This enables a suite of learning applications to use common services and shared data to glean insights 
about learners, content, context, and competencies within the 2018 TLA Test and Demonstration. 
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1.1 FUTURE LEARNING ECOSYSTEM 


Ecology is the branch of biology which studies the dynamic interactions between plants, animals, and 
micro-organisms and their environment, working together as a functional unit. Ecologies are living systems 
containing a diversity of factors that interact with each other and are self-organizing, adaptive, and fragile. 


The term ecology has been applied to the concept of learning as a metaphor for the many different tools, 
technologies, systems, and processes that interact with each other to promote knowledge. The term 
ecosystem is often used to describe all sources of learning that are available to employees within their 
organization; however, each organization has its own processes, contexts, relationships, and interactions 
that provides opportunities and resources for learning, development, and achievement. Further, the current 
landscape of institutional learning technologies consists of multiple data stovepipes even within the same 
organization. Training and education systems do not share information well with human resourcing and 
personnel systems; nor do they share data with operational systems that are the true indicators of 
organizational performance. 


The future learning ecosystem is defined by personalized and competency-based learning environments 
that promote interoperability and accessibility of learning resources across organizational boundaries and 
throughout an individual’s life. Today we understand that learning takes place both inside and outside the 
classroom. We live and work within a digital web of learning resources and learning experiences; each 
resource connects to others and feeds an overall structure in which learning takes place. At the heart of this 
ecosystem is the learner and the associated policies and processes used to track their learning experiences 
across the continuum of learning. 


The key to managing this lifelong learning data is the interoperability afforded through the technical 
standards, specifications, and practices that comprise the TLA research portfolio. These underpin the wide 
range of learning resources an individual will encounter and enable a federated data strategy that facilitates 
the sharing of this information with other systems within an organization and even across institutional 
boundaries. Conceptual interoperability also includes shared vocabularies, system architectures, 
frameworks, and reference models that help producers and consumers communicate effectively. The TLA 
research portfolio is grounded in a chain of research that helps the DoD optimize training and education 
resources, communicate information and knowledge anywhere and anytime, manage an overall data 
strategy, and enable greater flexibility for how and when learning should occur. This document is divided 
into the major sections outlined below. 


1.2 SPECIFICATIONS AND STANDARDS 


The future learning ecosystem powered by the TLA includes an interconnected, 
evolving community of learners, tools, systems, activities, and content that must all 
operate together. Making this shift requires the adoption of common technical 
standards across institutions, systems, and instructional content. 


Technical specifications and standards allow different learning resources to communicate with each other 
using a mutually agreeable common language. These standards form the fundamental building blocks for 
life-long learning by establishing consistent protocols that can be universally understood and adopted by 
any component within the future learning ecosystem to enable data exchange about learners, activities, and 
experiences. Beyond the interoperability standards required for integrating tools, learning experiences 
(activities and content), and a myriad of different data types, TLA research is also looking at shared 
vocabularies, linked data strategies, performance thresholds and an overall data strategy that ties them all 
together. 
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The 2018 TLA research investigated relevant standards, specifications, and policies that relate to metadata 
strategies, competency representation, activity tracking, Learner Profiles, and numerous communications 
protocols, and interface specifications. A specification/standard provides definitions, rules, roles, and 
responsibilities for data components within it. In many cases, a standard or specification will also define 
how and where data is stored or registered. Numerous specifications and standards were reviewed to 
investigate how they would influence the interoperable nature of the TLA. Candidate specifications were 
then implemented and tested within the 2018 TLA Reference Implementation. 


Within the TLA Reference Implementation, candidate standards and specifications are analyzed to 
determine whether the spec/standard is partially or entirely appropriate. If changes are required to extend 
the spec/standard, this work is documented to inform future standards/specification development bodies. It 
is important to note that the TLA is looking at specifications and standards used across other industrial 
sectors (e.g., finance, healthcare, manufacturing), across different communities (e.g., K-12 Education, 
colleges and universities, corporate learning) and for different purposes (e.g., machine to machine 
communication, evaluation/assessment). The 2018 research portfolio includes specifications and standards 
that relate to the following: 


e Interoperability standards: Characteristic definitions of a TLA component, activity providers, 
and services including performance thresholds, interface specifications, data models/schemas, etc. 


e Classification standards: Systematic groupings of data, components, systems, or services based 
on different characteristics or properties. 


e Standards for testing and evaluation: Characteristics and minimum thresholds measurements for 
security testing, regression testing, conformance testing, etc. 


e Standards for compliance and certification: Description of the functions and relationships 
between administration, curation, quality assurance, lifecycle maintenance, and governance of 
future learning ecosystem components, systems, and activities/content. 


With an ever-increasing amount of options, the task of selecting an approved standard or specification can 
be difficult. Each has their own advantages and drawbacks, and many have overlapping uses. The future 
growth of personalization services, data analytics, and integration with other business systems presents 
numerous challenges in achieving true interoperability between all the disparate data sources required for 
learning across an enterprise. The ADL Initiative will continue to evaluate different specifications and 
standards to determine how suitable they are for different aspects of the TLA. Section 2 of this document 
will describe technical standards and specifications evaluated for use in the 2018 TLA Reference 
Implementation. 


1.3 2018 TLA REFERENCE IMPLEMENTATION 


The goal of the 2018 TLA Reference Implementation is to evaluate candidate standards and specifications 
for potential inclusion within the TLA. In creating the 2018 TLA Reference Implementation, the ADL 
Initiative started with a foundational understanding of existing Information Technology (IT) components 
commonly utilized in Distributed Learning (DL) today. The reference implementation builds upon these 
components and in some instances, utilizes the same specifications and standards used by these components. 
While these existing specifications and standards are considered a starting point for the development of the 
TLA, the ADL Initiative is investigating additional specifications and standards to determine their 
applicability and suitability for the future learning ecosystem. 


3 The ADL Initiative — May FY19 


The 2018 TLA Reference Implementation maintained a few high-level considerations and design 
constraints. First, it was to be developed without proprietary data rights. The commercial products used 
within the system are open source. All vendor product communication was performed using industry 
standards. The system and user-action logs were maintained using the experience API. Systems 
communicated using a RESTful implementation. The system was deployed using Amazon Web Services 
(AWS) on virtualized server hardware, with purchased Platform and Backend as a Service (PaaS/BaaS). 


Figure 2 shows the major components of the TLA architecture as they were implemented in support of a 
TLA Test and Demonstration exercise at Fort Bragg in August 2018. This diagram establishes the context 
for how specifications and standards are currently being applied. The TLA components used in this 
implementation include a range of Technology Readiness Levels (TRLs) that impact the requirements and 
even the availability of relevant standards or specifications for certain components of the architecture. A 
primary goal of the TLA is to support plug and play interoperability of connected components, systems, 
content, and data. Each implementation of the TLA will utilize any variation of components that are 
required to meet the goals and objectives for that instance. Approved TLA components and associated 
specifications/standards will change over time as systems, capabilities, tools, and technologies continue to 
evolve. 
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Figure 2. 2018 TLA Reference Implementation: Logical View — Activity providers (green) are instrumented with 
XxAPI statements that are streamed to a Learner Record Store (LRS). The ADL Initiative services (blue) include the 
LRS, Authentication, Identity Management, TLA component discovery, and launch services. A competency 
management system (orange) collects xAPI assertions from a variety of assessments activities. These assertions are 
transformed into estimated levels of competency for each learner in the system. A Recommendation Engine (yellow) 
uses competency information and Learner Profiles to estimate mastery. The Learning Analytics Store collects 
information about all users in the System. Learner Inference information is also collected about each individual 
learner in the system. This information is used to augment a Collaborative Filtering algorithm with Content-based 
filtering that matches content to user characteristics. Learner Inference data was also written back to the Learner 
Profile that houses competency information. The recommender uses both algorithms to statistically rank each 
recommendation that was presented to the learner. For the 2018 Reference Implementation, the Learner Profile 
and the Activity Index were integrated into other TLA components. However, these will be separate components in 
future implementations. 
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Section 3 provides a detailed description of the specific components and architecture used in the 2018 TLA 
Reference Implementation. The data that flows between the different learning activities and the core TLA 
systems and components are areas where specifications and standards were applied in 2018. TLA 
components can generate terabytes of data in a typical course. Shared services between different 
components exchange data using a common communication protocol. These protocols are built using 
commonly available standards or specifications across separate web services. This allows the TLA to stay 
independent of vendors, products, or technologies and maintains focus on the interoperability between 
learning systems. 


A current snapshot of TLA components and services are shown in Table I and Table 2. Entries within these 
tables reflect current capabilities but will change as the TLA evolves. When instantiated in the 2018 TLA 
Reference Implementation, some TLA components were aggregated with other TLA components to create 
efficiencies or to facilitate the exploration of candidate specifications between components. Specifically, 
the Activity Index was instantiated as part of the recommendation engine and the TLA Learner Profile was 
instantiated as part of the Competency Management System. However, both components will be decoupled 
from those systems and will be set up separately in future TLA iterations. 


Table 1. 2018 TLA Components — The future learning ecosystem, powered by the TLA, contains several learning 
systems that must share data and work together to initiate, tailor, and track different learning experiences. This table 


lists major TLA components used in the 2018 TLA Reference Implementation. 


TLA Component 
Activity Index / 
Activity Registry 


| Short Description 


The Activity Index/Activity Registry manages the relationship between 
activities, competencies, enabling and terminal learning objectives, 
and other relevant information about each activity. It also contains 
access information and permission to allow access to activities. For 
2018, the Activity Index was part of the FLUENT architecture and in 
the future instantiations will be set up as a separate TLA component. 


Related Standards 
LRMI, Dublin Core 
Metadata, LOM 


Activity Streams 


These streams are time-stamped data about learner experiences 
tracked across learning activities and throughout the TLA. The 
Experience API (xAPI), version 1.0.3, is used to track performance 
inside TLA Learning Activities. 


xAPI, Caliper, 
HPML 


achievements, context, and other learner characteristics that may 
influence different TLA components. For 2018, the Learner Profile was 
part of the CaSS system. In future instantiations, the Learner Profile 
will be set up as a separate TLA component. 


Competency The framework is a model that broadly describes performance | ASN™, CASE™, 

Framework excellence within the TLA. Each framework includes numerous | O*Net, 
competency objects that may be applied to multiple occupational | Medbiquitous 
roles. 

Competency Competency Management System, CaSS, provides a common | RCD 

Management language/framework to describe competencies, formalizes the 

System mechanism for collecting evidence of attainment, and manages the 
lifecycle of learning. 

Credential Credentials are learning artifacts gained through an organization | CDTL, Open Badges 

Management based on merit. CaSS is the backend to Credential Engine. For the 2018 | 2.0, Verifiable 

System implementation, CaSS was used for both competencies and | Claims 
credentials. 

Data Dashboards, | These components process data to provide insight and reports. | xAPI, TBD 

Data Analytics, Currently connected to LRS functionality. In future implementations, 

and they will pull data from different systems to mashup different data for 

Visualizations different users. 

Learner Profile Learner Profiles store learner data about competencies, | AIS, CLR 
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TLA Component 
Learner Record 
Provider (LRP) 


| Short Description 

TLA learning activities include videos, PDF documents, SCORM 
courses, serious gaming applications, e-books, mobile learning 
content, concept maps, and multiple different types of assessments. 


Related Standards 
xAPI 


Learner Record 


An LRS, Learning Locker, stores xAPI statements across a wide variety 


xAPI, Caliper, 


System (LMS) 


tracking, and assessment of student work. 


Store (LRS) of learning activities, experiences, devices, and platforms. HPML 
Learning An LMS delivers and manages instructional content, and typically | SCORM 
Management handles student registration, online course administration, and 


Recommendation 
Engine 


FLUENT — Utilizes data from the LRS, Learner Profile, Activity Registry, 
and metadata attached to learning content to adaptively sequence 
the delivery of different learning activities to each individual. 


AIS (Too early to 
evaluate) 


Single Sign-On 


For the purpose of the 2018 TLA Reference Implementation, the ADL 
Initiative staff utilized Keycloak™ for identity and access management. 


OIDC, OAuth 2.0, 
SAML 2.0 


This capability allows a user to log-in once and have access to multiple 
applications within the TLA ecosystem. This was also instrumental in 
establishing universal unique identifiers for each learner to protect 
each learner’s true identity. 


Table 2. 2018 TLA Services — This table lists the underlying services used to enable communications between the 
different TLA components to communicate with each other. 


TLA Services Short Description Related Standards / 
Technologies 


Assessment Within the 2018 TLA Reference Implementation, CaSS utilized a QTI, AICC, SCORM, IEEE 
component called an Evidence Mapper. This collected assessment 1484.11, xAPI 

evidence and other data used to infer competency for a specific skill. 
Authorization is the access granted by an authority to specific 


Authorization Keycloak™, CAC, web3, 


activities/components OAuth 2.0 
Authentication Authentication is the means by which identity is validated with the Keycloak™, CAC, web3, 
purpose of access OIDC 
Content This has not currently been implemented within the 2018 TLA  OAI/PMH, SKOS 
Discovery Reference Implementation. But it is being considered for future 
iterations. This service will be closely tied to metadata generation. 
Identity Identity management is a coordination of services and protocols for Keycloak™, CAC, web3, 
Management preserving and representing a learner’s uniqueness across the system. OIDC, UUID 
Launch Launch refers to being responsible for delivery and session Cmi5, WebSockets 
management of a resource. 
Messaging Within the 2018 TLA Reference Implementation, numerous messaging http/s, Websockets, 
services were implemented to push data from one component to AMQP 
another. 
Metadata This capability was manually created for all content used in the 2018 | Schema.org 
generation TLA Reference Implementation. Research into automation tools, Vocabularies, LRMI, 


techniques, and technologies is expected to inform future iterations. | OASIS CMIS, LOM, 
Dublin Core Metadata 
W3C Push Notifications 
API, WhatWG 


Notifications API 


Notifications Notifications is not currently implemented within the 2018 TLA 


Reference Implementation but being considered for future iterations. 


Search Search is not currently implemented within the 2018 TLA Reference SPARQL, PLQL, 
Implementation but being considered for future iterations. ElasticSearch 

TLA Service For 2018, this service exposed a RESTful API for either manual or XRDS, REST, WS- 

Registry, aka dynamic IP address registration for all major components and Discovery, RAMLET 

Discovery services. 


6 The ADL Initiative — May FY19 


1.42018 TLA TEST AND DEMONSTRATION 


The TLA Test and Demonstration was held in August of 2018 to assess the 2018 TLA Reference 
Implementation. The 2018 TLA Reference Implementation was developed by a cross-organizational team 
to evaluate the various software services, technical components, and learning activities, all of which 
exchanged data using a candidate set of technical standards and specifications. The 2018 TLA Test and 
Demonstration provided hands-on experience with the 2018 TLA Reference Implementation by students, 
instructors and administrators. Additional data was collected as content was tailored and instrumented to 
work within the reference implementation. 


A primary goal of this study was to better understand what the impediments to full adoption and/or diffusion 
of the TLA might be with the current design. Another goal was to assess the practical functionality of the 
system and the viability of the candidate specifications used in the 2018 TLA Reference Implementation. 
Numerous communications protocols, interface specifications, and data models (schemas) are required to 
facilitate the systems to systems interoperability required by the TLA. Many of these candidate 
specifications required modifications or custom extensions to meet the requirements for delivering the 
myriad of different learning activities. The 2018 TLA Test and Demonstration included traditional e- 
learning content, a browser-based concept-map assessment application, an e-book, micro-learning activities, 


a computer-based serious game, and physical (non-digital) instructor-led learning activities. 
= 


Figure 3. 2018 TLA Test and Demonstration - The TLA was tested in collaboration with the John F. Kennedy 
Special Warfare Center and School (JFKSWCS), Fort Bragg, NC. 


The 2018 TLA Test and Demonstration showed adaptation across these different learning activities, but 
within the context of a single course. Future implementations of the TLA will provide the ability to adapt 
across learning trajectories, careers, and experiences. The 2018 TLA Test and Demonstration used learning 
content related to an existing course entitled Combat Observation and Decision-Making in Irregular and 
Ambiguous Conflicts (CODIAC). As shown in Figure 3, it was tested by approximately 60 participants 
from the Army Special Warfare Education Group (Airborne) (SWEG(A)) at the John F. Kennedy Special 
Warfare Center and School (JFKSWCS). Testing consisted of users interacting with the CODIAC content 
for 14 hours over five days, with data collected before, during, and after. 


Automated data collection efforts at the event produced a significant quantity of data. The system collected 
over 450,000 xAPI statements during the 4-day event, which were later anonymized and distributed for use 
ina TLA hack-a-thon held at the 2018 iFest. Other than xAPI statements, the system captured each learner’s 
final competency profile, their activity history, and statistically significant pre-test and post-test 
differentials. Additionally, the surveys and focus groups collected learner opinions about usability, 
relevance, and other impressions from various users of the reference implementation. Section 4 of this 
document provides an overview of the 2018 TLA Test and Demonstration. 
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1.5 2018 DATA ANALYTICS AND VISUALIZATION 


The goal of the hack-a-thon was to present opportunities for a broad community of experts to explore the 
TLA in depth and discuss new or alternative methods to combine and process learning data to empower 
analytics. The hack-a-thon investigated the potential of learning analytics, identified new ways to visualize 
collected data, and exposed gaps in the practical use of the Experience API (xAPI) across different types 
of media. The ADL Initiative provided a large set of TLA data from the 2018 TLA Test and Demonstration 
for participants to aggregate, analyze, and visualize. High-level goals presented at the beginning of the 
hack-a-thon included the following: 


e Make multi-activity time-series data accessible to a non-data scientist. 

e Correlate activities and outcomes for issuing awards, badges, and credentials (micro- 
credentials). 

e Create visualizations from the existing data set based on content, anomaly detection, or 
other options. 

e Identify publicly available data and third-party APIs that can be instrumented with xAPI 
to drive analytics. 


Hack-a-thon attendees came from a variety of organizations spanning industry, academia, and government. 
Most participants identified as software developers or data scientists and experience levels were mixed. 
Participants were organized into groups based on level of experience and focus area. Participants had the 
opportunity to network with experienced developers, educators, and data scientists to learn best practices 
for implementing, structuring, storing, and processing xAPI data and other data generated by TLA 
components at scale. Section 5 of this document presents and discusses these results. 
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2. TLA SPECIFICATIONS AND STANDARDS 


Research is continually performed to evaluate how standards and specifications might be utilized across the 
future learning ecosystem, at various stages of a career trajectory, and across institutional boundaries. Table 
3 below provides a mapping of relevant specifications and standards for each TLA component or service. 
This table summarizes the candidate standards and specifications utilized or investigated for 2018. The 
following sections provide an in-depth synopsis of how each specification or standard was investigated, 
utilized, and/or modified to support the 2018 TLA Reference Implementation. 


Table 3. Summary of TLA Specifications — This table breaks down each specification to show how and where it 
was used. The specifications are grouped and listed according to the TLA component it is aligned with. There is an 
expectation that TLA Components and their outputs adhere to the referenced specifications. Supporting vocabularies, 
rules, and software ensure this process. 


TLA Component Standard/Specification Data Store/Registry 


Activity Registry LRMI Json Activity Index 
IEEE 1484.12.1 LOM XML Activity Index 
Dublin Core Metadata XML, RDF Activity Index 
Schema.org Vocabularies Activity Index 
Activity Stream IEEE xAPI http/https with json payloads | Learner Record Store 
IMS Global Caliper http/https with json-LD Learner Record Store 
payloads 
W3C Activity Streams 2.0 = Json, json-LD Needs Research 
SISO HPML Human Performance Data 
Model 
Competency ASN™ CaSS 
Management CASE™ CaSS 
O*Net CaSS 
Medbiquitous CaSS 
RCD / RCDEO CaSS 
Others CaSS 
Credentials CTDL Credential Authoring Credential Registry 
Language 


IMS Global Open Badge 
W3C Verifiable Claims 


Data Analytics and Multiple DAVE Algorithms, DAVE Dashboards, TLA Portal 
Visualization Visualizations, Data Cards 

Environment 

Learner Profile CaSS Proprietary 


Airmen Learner Record 
TBD Research 


Learner Service TBD Research 

Recommender AIS Needs Research N/A 

Identify Management = Keycloak™ OpenID Connect (profile of TLA Common Training 
/ Single Sign On OAuth2.0) and Education Portal 
Learner Record ePUB 3 (formerly IEEE 

Provider - eBooks ADB), xAPI 


The following sections describe the candidate specifications and standards used for each TLA component 
or service, providing a thorough evaluation of capabilities and recommended extensions to support TLA 
requirements with additional insights describing how they complement or compete with other related 
specifications. 
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2.1 ACTIVITY REGISTRY 


The Activity Registry includes an Activity Index that stores metadata about each TLA learning activity. 
The approach used in the 2018 Reference Implementation relied heavily on the Learning Resource 
Metadata Initiative (LRMD specification. While LRMI shows promise for tagging new content, there are 
thousands of learning resources tagged using metadata schemas managed under different organizations. 
Given the current state of technology, the TLA needs to be compatible with many of the more common 
metadata formats. 


All data formats currently being investigated only support human-readable descriptions and don’t consider 
the type of metadata generated by modern machine learning algorithms. These systems generate metadata 
based on a statistical analysis to describe concepts such as engagement, effectiveness, or any number of 
other descriptors that would be useful to other TLA components. In the future, an Activity Registry will 
include data and connected services that discover, catalog, and store information about TLA compatible 
learning activities. A primary goal of the 2019 research is to understand how the Activity Registry drives a 
Common Course Catalog, populated by different stakeholders. Other research slated for 2019 includes the 
identification of approaches for integrating paradata into the Activity Registry. The following metadata 
standards are being looked at to describe TLA content. 


2.1.1 Learning Resource Metadata Initiative (LRMI) 


The LRMI’ is a common metadata framework developed by Creative Commons (CC) and the Association 
of Educational Publishers (AEP) for describing and tagging of educational resources in web-based 
instruction and training. The LRMI metadata schema was adopted by Schema.org in April 2013. This 
allows anyone who publishes or curates educational content to use LRMI markup to provide rich, 
education-specific metadata about their resources with the confidence that this metadata will be recognized 
by major search engines. 


The LRMI specification is a collection of classes and properties for metadata markup and description of 
educational resources. The specification provides clear guidance of the terms within and how to use them 
both in coding and through descriptive language to be followed by those implementing it. The attributes 
defined by the metadata are clear and the rules surrounding the specification drive interoperability. LRMI 
1.1 specification is very stable and has seen significant adoption. Within the context of the 2018 TLA 
Reference Implementation, LRMI was selected because it provided the most contextually relevant data 
fields to empower the FLUENT recommender. 


The LRMI specification includes an AlignmentObject as part of the specification. This object 
describes an alignment between a learning resource and a node in an educational framework. This feature 
was used extensively in the 2018 TLA Reference Implementation. The 2018 alignments included 
educational audience, educational use, competency object, simple or complex, interactivity level, and others. 
While limited, these use cases proved the ability of the AlignmentObject to serve its intended purpose. 
The 2019 research effort needs to build out a minimum set of metadata requirements that should be 
implemented for any future learning activities or content. 


' http://Irmi.dublincore.org/ 
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2.1.2 IEEE 1484.12.1 Learning Object Model (LOM) 


As shown below in Figure 4, Learning Object Model (LOM) provides a broad framework for describing 
learning objects to facilitate search, evaluation, acquisition, and use. LOM is used by the SCORM reference 
model to provide descriptive information about learning objects, including the title, author, description, 
keywords, educational objective, and other relevant information. The LOM data model specifies which 
aspects of a learning object should be described and what vocabularies may be used for these descriptions; 
it also defines how this data model can be amended by additions or constraints. The metadata is hierarchical, 
starting with a root and expanding to leaf nodes. Alignment of an object can be connected to a discipline, 
idea, prerequisite, educational objective, accessibility, restrictions, educational level, skill level, security 
level, or competency. 


LOM v1.0 schema 


Description [0..*}: Langstring 
Keyword [0..*]; Langstring 


InstallationRemarks [0..*]: Langstring 


InteractivityLevel [0..1]: Enumerated 


OtherPlatform Requirements (0..*)" Langstring 


SemanticDensity [0..1]: Enumerated 


; 0..* 
Resource 0.1 7. Relation 9. Classification 
Description [0..*]: Langstring Kind [0..1]: State Purpose [0..1): State 
0..* Description [0..1]: Langstring 
7 Keyword {0..*]: Langstring 
8. Annotation em 0° 
Entity (0..1]: CharacterString TaxonPath Taxon 
Date[0..1] DateTime Id [0..1]: CharacterString 
Description [0..1] Langstring Source [0..1]: Langstring Entry [0..1]: Langstring 
0.1 
at [ztfecycle | 4. Technical oo we 
1. General 2. Lifecycle Farimak (0:4: araclorStieng 5. Educational 6. Rights 
Title [0..1]: Langstring Version [0..1]: Langstring Size [0..1]: CharacterString InteractivityType [0..1]: State Cost [0..1]: State 
Language [0..*]: CharacterString Status [0..1]: State Location [0..*}: CharacterString LearningResourceType [0..*]: State Copyright and other restrictions [0..1]: State 


Description {0..1]: Langstring 


Coverage [0..*]" Langstring 
Structure [1]: State 
Aggregation Level [1]: Enumerated 


Duration (0..1]: Duration IntendedEndUserRole [0..*)" State 
Context [0..*): State 


Difficulty [0..1]: Enumerated 


ae 
TypicalLearningTime [0..*]: Duration 
Description (0..*]: Langstring 
Language [0..*]: CharacterString 


3. Meta-Metadata 


MetadataSchema (0..1]: CharacterString 


orComposite 
Type [0..1]: State | 


o* 
Catalog [0..1]: Langstring 
Entry [0..1]: Langstring 


Figure 4. Learning Object Metadata — LOM comprises a hierarchy of elements that includes 9 categories, each of 
which contains sub-elements. The semantics of an element are determined by its context: they are affected by the 
parent or container element in the hierarchy and by other elements in the same container. 
https://commons.wikimedia.org/w/index.php ?curid=16026109 


Contribute 
Role [0..1]: State 
Entity [0..*]: CharacterString 
Date [0..1]: DateTime 


Name [0..1]: State 
MinimumvVersion (0..1]: CharacterString 
MaximumVersion [0..1]: CharacterString 


The purpose of this Standard is to allow the creation of LOM instances in XML. This allows for 
interoperability and the exchange of LOM XML instances between various systems. When implementing 
the LOM as a data or service provider, it is not necessary to support all the elements in the data model, nor 
need the LOM data model limit the information which may be provided. The creation of an application 
profile allows a community of users to specify which elements and vocabularies they will use. Elements 
from the LOM may be dropped and elements from other metadata schemas may be brought in; likewise, 
the vocabularies in the LOM may be supplemented with values appropriate to that community. For example, 
the Healthcare LOM’, developed by the Medbiquitous consortium extends the LOM standard and provides 
custom vocabularies for some metadata elements. 


* https://ieeexplore.ieee.org/document/1032843/ 
3 https://www.medbiq.org/working_groups/learning_objects/HealthcareLOMSpecification.pdf 
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The attributes defined by the metadata are clear and the rules surrounding the specification drive 
interoperability. The LOM specification (1484.12.1-2002) is very stable and has seen significant adoption. 
The ADL Initiative stakeholders have thousands of SCORM courses and other content that has been 
encoded with LOM metadata so any TLA component that relies on metadata needs to be compatible with 
LOM. As with most metadata formats, the consistency of quality metadata is a known issue. 


2.1.3 Dublin Core Metadata Initiative (DCMI) Metadata 


The Dublin Core Schema’ is a small set of vocabulary terms that describe digital resources (video, images, 
web pages, etc.), as well as physical resources such as books or CDs, and objects like artworks. The Dublin 
Core Metadata Element Set is a vocabulary of fifteen properties for use in resource description. It is part of 
a larger set of vocabularies and technical specifications maintained by the DCMI. Dublin Core metadata 
may be used for multiple purposes, from simple resource description to combining metadata vocabularies 
of different metadata standards, to providing interoperability for metadata vocabularies in the linked data 
cloud and Semantic Web implementations. 


The full set of vocabularies also includes sets of resource classes, vocabulary encoding schemes, and syntax 
encoding schemes. The terms in DCMI vocabularies are intended to be used in combination with terms 
from other, compatible vocabularies in the context of application profiles and the DCMI Abstract Model. 
As part of an extended set of DCMI Metadata Terms, Dublin Core became one of most popular vocabularies 
for use with RDF, more recently in the context of the Linked Data movement. The DCMI assumed 
stewardship of the LRMI specification in 2014. The Dublin Core schema is relevant because it is widely 
used across the Internet. While the educational alignment aspects of DCMI are not as robust as LOM or 
LRMI, the ability to tie into additional vocabularies provides an attractive mechanism to identify and 
catalog content from different communities of interest, industries, or demographics. 


2.1.4 Schema.org Vocabularies 


Founded by Google, Microsoft, Yahoo and Yandex, Schema.org vocabularies are developed by an open 
community with a mission to create, maintain, and promote schemas for structured data on the Internet. 
Each schema.org item type has its own set of descriptive properties. The broadest item type is Thing, which 
has four properties (name, description, URL, image). More specific types share properties with broader 
types. For example, a Place, Person, or CreativeWork is a more specific type of Thing. 


LRMI’s adoption into Schema.org vocabularies provides many benefits. In theory, nearly any schema.org 
Thing could be a learning resource. Therefore, LRMI addresses those metadata properties that distinguish 
content when it is deliberately used for learning. This was done by adding learning-resource properties to 
key root types. Currently, CreativeWork properties include educationalUse and 
educationalAlignment. A more specific type of CreativeWork includes a Course® which may be 
offered as distinct instances at which take place at different times or take place at different locations or be 
offered through different media or modes of study. An educational course is a sequence of one or more 
educational events and/or other types of CreativeWork which aims to build knowledge, competence or 
ability of learners. 


As shown in Figure 5, a learning activity is a schema.org>Thing>CreativeWork that inherits the 
properties that every schema.org Thing has and can be used to support multiple forms of alignment that are 


4 http://www.dublincore.org/specifications/ 
> https://schema.org/ 
® https://schema.org/Course 
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possible between a resource and an educational framework. The AlignmentObject can also be used 
to distinguish between resources that teach and assess. This presents the ability to collect a substantial 
corpus of paradata about how different types of content apply to different instructional domains and enables 
new insights into which content is more effective. 


text 


alignment ramework name 


Type educational 
Framework 


@ Ttarget URL 
_educational 
Alignment 


Creative Work Alignment Object 


Educational 
Framework 


Figure 5. Alignment between a Schema.org CreativeWork and a Node in an Educational Framework’ — The 
2018 TLA Reference Implementation used LRMI’s AlignmentObject to reference a competency framework that 
provided a structured description of required knowledge, skills, abilities, and their interrelated relationships. 


2.2 ACTIVITY STREAMS 


The future learning ecosystem will offer a wide range of learning content across the continuum of learning. 
One learner may spend time reading technical articles or writing a blog post while another interacts with 
video content and interactive exercises. Activity Streams are a list of events generated by individuals, 
groups, applications, or learning activities that provide details about the ongoing experiences to other TLA 
components. 


The types and variety of activities that are used for learning can often be associated with a specific delivery 
modality. Instructor-led classroom training will create one set of instructional activities, while serious 
games and simulations have the potential of generating a completely different set of activities. This has 
potential to have two similarly named activities with two different contexts for how those activities are 
being applied and the experiences they encompass. A common vocabulary is necessary to ensure all 
learning activities across different communities accurately describe the experience. By formalizing this 
vocabulary, a set of attributes and rules about the data is established such as how they are stored, retrieved 
and accessed by other components, systems or activities. 


The different activity stream specifications investigated for inclusion in the TLA are similarly structured. 
Each specification includes serialized data streams that consist of statements about activities. Such 
statements typically involve a subject (the person doing the activity), a verb (what the person is doing), and 
a direct object (what the activity is being done to or with). The subject of an activity is nearly always the 
learner but could foreseeably be an instructor, cohort, or other. The direct object of an activity is presented 
differently depending on its context. Verbs need to conform to a common vocabulary. Otherwise different 
organizations will use different verbs to describe the same activity or the same verb to describe different 
activities. 


T https://blogs.pjjk.net/phil/explaining-the-Irmi-alignment-object/ 
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2.2.1 Experience API (xAPI) 


The xAPI° specification is in the process of becoming a standard through the Institute of Electrical and 
Electronics Engineers — Learning Technology Standards Committee (IEEE-LTSC)’. The xAPI specifies a 
structure to describe learning experiences and defines how these descriptions can be exchanged 
electronically. The main components of xAPI are the data structure called Statements and the data 
storage/retrieval capability called the Learning Record Store (LRS). The xAPI specification has stringent 
requirements on the structure of this data and the capabilities of the LRS. Statements are data triples that 
use an actor, a verb, and an object to describe any experience. Each statement also includes timestamps and 
unique, resolvable identifiers. The transport is HTTP/HTTPS with JSON payloads. 


Any enabled device can send xAPI statements including mobile phones, CPR dummies, serious games and 


°10 offers acommon 


simulations, and any number of other learning systems. The “xAPI Profile Specification 
way to express controlled vocabularies across these different mediums, provides instructions on the 
formation of xAPI statements, and describes patterns of xAPI usage that provides additional context to a 
domain, device, or system. The xAPI Profiles Specification also adds tools to support authoring, 


management, discovery and/or adoption, including additional data elements and properties. 


A Learning Record Store (LRS) is the implementation of the server-side requirements associated with the 
xAPI specification. The LRS is a key component of the xAPI architecture. It is the application interface for 
storing, accessing, and often visualizing the data about learning experiences, activities, and performance. 
The LRS is essentially a web service that leverages REST to ensure all data from xAPI can be stored and 
retrieved. It can share data and transcripts with other LRSs so learning experiences can follow you from 
one organization’s LRS to another. An LRS could be optionally integrated with any application such as an 
LMS, human resources (HR) system, or it could serve as centralized data store in an enterprise learning 
ecosystem. Third party applications which send or retrieve learning activity data will interact with the LRS 
as the data store for xAPI data. From this perspective, an LRS is also required to validate the format of the 
statements as many of the requirements in the xAPI specification are targeted toward the LRS component. 


2.2.2 IMS Global Caliper 


The Caliper Analytics® Specification’! defines a set of concepts, relationships, and rules for describing 
learning activities. Each activity domain is described in a metric profile that models a learning activity or 
any supporting activities. Each profile provides a shared vocabulary that developers use to describe user 
interactions in a consistent manner. Each profile is composed of one or more Event types (e.g., 


AssessmentEvent, NavigationEvent). Each Event type is associated with a set of actions 
undertaken by learners, instructors, and others. A Caliper Event describes the purposeful action undertaken 
by the actor within a given learning context. The data model is based upon the (Actor, Action, 
Activity) triple. Various Entity types representing people, groups, and resources are provided to 
describe the relationships established between participating entities and the contextual elements relevant to 
the interaction (e.g., Assessment, Attempt, CourseSection, Person). 


Storage or data retrieval capabilities are not defined as a part of the specification; however, the defined data 
attributes about Events and Entities are clear and the rules surrounding the specification drive 


8 https://github.com/adInet/x A PI-Spec 

* https://www.tagxapi.org/ 

10 https://github.com/adInet/xapi-profiles 

"| https://www.imsglobal.org/sites/default/files/caliper/v 1p 1/caliper-spec-v1p1/caliper-spec-v1p1.html 
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interoperability. It is comparable to the xAPI specification but offers a more controlled vocabulary. It does 
implement ideal behaviors that xAPI has identified for future profiles. The Caliper specification appears to 
be gaining adoption across learning tool providers including Blackboard, Canvas, and others. Because other 
IMS Global standards are prevalent in K-12 education, it is anticipated that adoption will continue to grow. 


2.2.3 W3C Activity Streams 2.0 


The W3C Activity Streams” is an open format specification for activity stream protocols, which are used 
to syndicate activities in social web applications and services, like those in Facebook or LinkedIn. An 
Activity is a semantic description of an action. This specification describes a JSON-based serialization 
syntax for the activity vocabulary that conforms to a subset of [JSON-LD] syntax constraints but does not 
require JSON-LD processing. The Activity Streams 2.0 Vocabulary defines a set of abstract types and 
properties that describe past, present, and future activities. For example, the activity vocabulary defines five 
specific types of actors that include Application, Group, Organization, Person, and Service. 


Every JSON object in an Activity Streams 2.0 document is either an object or a link. The object is the 
primary base type for the Activity Streams 2.0 Vocabulary. In addition to having a global identifier 
(expressed as an absolute Internationalized Resource Identifiers (IRD) using the id property) and an object 
type (expressed using the type property), all instances of the object type share a common set of properties 
normatively defined by the activity vocabulary. Object types are segmented into a set of eight core types, 
but external vocabularies can be used to express additional detail not covered by the activity vocabulary. 


This specification presents opportunities for future TLA implementations. The W3C specification is already 
in use across many social platforms that are already part of the future learning ecosystem. Additional value 
may also be found in the way this specification implements properties and their ability to push/pull 
metadata about different objects referenced in the stream (e.g., information about places, content, other 
people). The capacity to include content metadata as a payload presents interesting possibilities in the way 
we catalog learning experiences and could have a huge impact on the way a competency management 
system makes inferences about learner competencies after interacting with a TLA learning activity. 


2.2.4 SISO Human Performance Markup Language (HPML) 


HPML!? is an XML-Schema-based language intended to support trainers, instructors, leaders and others in 
the evaluation of individuals and teams as they perform their job functions. HPML provides schemas for 
organizing the information relevant to defining performance measurements, including computations, 
measures, assessments, results, and instances/periods. Specifically, it is an XML based language designed 
to express performance measurement concepts in a format that is both machine and human readable. 


HPML enables the explicit combination and transformation of performance data into performance 
measurements and assessments. These are decoupled from a specific training system but can be used to 
specify the data elements to be collected from a system, the calculations to use to process the data, and 
when to produce performance measurement results. As shown in Figure 6, HPML provides a flexible 
framework for configuring and executing performance measurement and assessments. The schema is 
separated into six distinct groups that make up the core components of HPML and can be added to or 
expanded with additional links in the schemas. 


2 https://www.w3.org/TR/activitystreams-core/#documents 
13 https://discussions.sisostds.org/index.htm?A0=S AC-PDG-HPML 
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Figure 6. HPML High-Level Conceptual Architecture — HPML includes a Human Performance Data Model that 
defines a set of rules that dictate how performance data is measures, computed, and assessed. It is agnostic about 
the kinds of data used to compute the measurements and assessments. 


HPML as a specification includes both the markup language used to author the activity stream and the 
Human Performance Data Model used to measure and assess performance based on the incoming stream 
of data. This is especially useful when considering approaches for collecting performance data out of live, 
virtual, and constructive simulations, but also provides immediate value to the TLA in formalizing a method 
for rolling up future learning ecosystem activities into meaningful assessments. More specifically, by 
externalizing the measurement and assessment specifications, these can be used with any data source (e.g., 
other activity streams). 


2.3 COMPETENCY MANAGEMENT 


Competency management requires the generation of rich and traceable data about learning experiences, 
how they relate to skill proficiency, and the knowledge, skills, abilities, and behaviors that individuals need 
to do their job. Competencies describe specifically what people need to do to be effective in their roles, and 
it clearly establishes how their roles relate to organizational goals and success. Each individual role has its 
own set of competencies needed to perform the job effectively. 


Competency-based learning (CBL) represents a transition from curricula focused on abstract knowledge 
and pursuit of certificates to curricula built around authoritative performance indicators that guide learning 
experiences based on challenges in the workplace that unlock human potential. Proficiency is a complex 
and critical concept that requires relevant, trusted data that can be used as evidence about an individual’s 
mastery against a specific set of competencies. A competency framework is a structure for defining the 
knowledge, skills, abilities and attitudes (or other characteristics) required to do a job. Competency 
frameworks are widely used in business for defining and assessing competencies within organizations in 
both hard and soft skills that jointly define successful job performance. There are numerous competency 
frameworks available and numerous specifications that drive them. Some of these competency 
specifications are included later in this section. 
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2.3.1 Achievement Standards Network™ (ASNTM) 


ASN™ provides access to machine-readable representations of learning objectives and curriculum 
standards". It provides an RDF-based framework based on the Dublin Core Metadata Initiative’s (DCMI) 
syntax-independent abstract information model (DCAM). The DCAM is intended to support development 
of Dublin Core Application Profiles (DCAP) of which the ASN DCAP is an example. The ASN™ 
framework is made up of standards documents, which represent the overall competency framework; and 
Statements, that represent the individual achievements within the overall framework. A set of core 
properties define the relationships between the two in terms of an Entity Relationship model. Structural 
relationships replicate the relationships between different components of the Standards Document and 
semantic relationships define the relationships of meaning between statements (e.g., assertion of 
equivalence). 


ASN™ is designed for interoperability and open access to learning objectives. It has seen wide adoption in 
the K-12 community. The ASN Description Framework (ASN-DF) also provides the means to create ASN 
profiles through inclusion of additional properties and classes from other namespaces and definition of 
constraints on data patterns. ASN-DF provides a small vocabulary of classes and properties and a set of 
mechanisms for tailoring an ASN profile based on the Dublin Core's conceptualization of application 
profiles and description set templating as well as refinements to these conceptualizations developed by the 
U.S. Library of Congress to support BIBFRAME profiling. 


2.3.2 Competencies and Academic Standards Exchange™ (CASE™) 


The IMS Global Competencies created the CASE™ specification'> to define how systems exchange and 
manage information about learning standards and competencies in a consistent and digitally-referenceable 
way. CASE™ connects standards and competencies to performance criteria and provides a way to transmit 
rubrics between various platforms. Within CASE™, the underlying structure of both a competency and an 
academic standard are represented using the same data model. The data model is composed of three core 
constructs: 


e ©The root definition document - the CFDocument is the top-level structure that holds the set of 
statements that define the individual competencies/academic standards. 


e The set of composition statements - the CF Item is the set of statements into which the top-level 
competency/academic standards have been decomposed. 


e The rubric—the CFRubric defines how mastery of the competency or standard can be achieved. 
This requires the definition of specific criteria used for each of the scores that can be awarded 
during an assessment. 


The CASE™ specification defines internal relationships between Parent/Children statements like , 


Precedes, IsRelatedTo, Exemplar, and IsPartOf. All competency frameworks published in 


CASE™ format can be linked together in a network of equivalent or aligned competencies. Having 
universal identifiers for each competency makes it possible for any tools or applications to share 
information between systems. 


'4 http://www.achievementstandards.org/ 
'S http://www.imsglobal.org/activity/case 
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2.3.3 O* Net 


The Occupational Information Network (O*NET)!° is sponsored by the U.S. Department of Labor, 
Employment and Training Administration (USDOL/ETA) to facilitate and maintain a skilled workforce. 
Central to the project is the O*NET database, which contains hundreds of standardized and occupation- 
specific descriptors on almost 1,000 occupations covering the entire U.S. economy. Every occupation 
requires a different mix of knowledge, skills, and abilities, and is performed using a variety of activities 
and tasks. As shown in Figure 7, the O*NET database identifies, defines, describes, and classifies 
occupations through Experience Requirements (Training, Licensing), Worker Requirements (Basic and 
Cross-Functional Skills), Occupation Requirements (Work Activities, Context), Worker Characteristics, 
Occupation Specific Information, and Occupational Characteristics. 


Worker-oriented 


Worker Worker Experience 
Characteristics Requirements Requirements 


Abilities Skills * Knowledge Experience and Training 
Occupational Interests Education Skills - Entry Requirement 
Work Values Licensing 
Work Styles 


Cross Occupation 


Occupation Specific 


Occupational ; = 
Requirements Olefe Fler ilelgmn) erclellilel 


Work Tai ielgear-leleya 
Work Activities: Bepeeeees 


General « Intermediate + Detailed Characteristics 


Organizational Context 
Work Context 


Title * Description 
Alternate Titles 


Labor Market Information Tasks 
Occupational Outlook Tools and Technology 


Moaiiiw LY 


Job-oriented 


Figure 7. O*NET Content Model — This model provides the framework that identifies and organizes the 
distinguishing characteristics of an occupation. The model defines the key features of an occupation as a standardized, 
measurable set of variables called descriptors. This hierarchical model starts with six domains that expand to 277 
descriptors. 


The value of the O*NET Database to the TLA is in leveraging the existing resources these government 
sponsored activities have already amassed. Their military transition search connects occupations in the 
O*NET database to other classifications systems within the military'’. This is accomplished through data 
from the Military Occupational Classification (MOC) system at the Defense Manpower Data Center 
(DMDC). This capability is also available via web services. Other resources include Careers in the Military 
and links to the COOL projects for Army, Navy, Marine Corps, and Air Force. 


16 https://www.onetonline.org/ 
17 https://www.onetcenter.org/crosswalks.html 
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2.3.4 MedBiquitous Competency Framework 


The MedBiquitous Competency Framework!*, ANSI /MEDBIQ CF.10.1-2012, is a technical standard for 
representing competency frameworks in XML, accredited by the American National Standards Institute. 
Organizations that publish competency frameworks can do so in this standard format, making it easier to 
integrate competency frameworks into educational technologies like curriculum management systems. The 
standard allows medical schools and other health professions schools to connect their curriculum, learning 
resources, and assessment data back to a common set of competencies, ultimately enabling competency- 
based views of the curriculum and of learner performance. The data model establishes relationships between 
competency objects that narrow or broaden the definition of overall competency. 


The MedBiquitous Competency Object specification provides a consistent format and data structure for 
defining a competency object, an abstract statement of learning or performance expectations, and 
information related to the statement. Statements can be learning outcomes, competencies, learning 
objectives, professional roles, topics, classifications/collections, etc. The competency object may include 
additional data to expand on or support the statement. This specification is meant to be used with the 
MedBiquitous Competency Framework specification. 


2.3.5 IEEE 1484.20.1 Reusable Competency Definitions (RCD) 


The current RCD standard’? defines a data model for describing, referencing, and sharing competency 
definitions, primarily in the context of online and distributed learning. The standard provides a way to 
represent formally the key characteristics of a competency, independently of its use in any specific context. 
It enables interoperability among learning systems that deal with competency information by providing a 
means for them to refer to common definitions with common meanings. This specification enables the 
storage of a competency framework in the form of a Dynamic Acyclic Graph (DAG) where competency 
objects define the knowledge, skills, conditions and other factors that role up into an ability to do a job. The 
edges between each competency object inform the relationship between them and have the potential to 
identify whether they’re dependent, complimentary, conflicting, or other. This provides an extensible 
mathematical underpinning to a competency framework that accommodates the relationships defined in 
any other foreseeable competency framework. 


The Data Model for RCD Working Group (WG) 20° was put together in September 2018 to revise the 
2008 standard. The Competencies WG 20 intends to take the Credential Ecosystem Mapping Project’s 
mapping of competencies metadata and update RCD to represent the most common elements that are found 
in multiple standards addressing competencies and competency frameworks. The ADL Initiative will 
monitor this WG for progress. 


2.4 CREDENTIALS 


A credential is a testament of qualification and competence issued to an individual by an authoritative third 
party. Examples of credentials include academic diplomas, degrees, certifications, professional licenses, or 
other proof of occupational qualification. Within the TLA, credentials are an exchange format by a trusted 
party that formally encapsulates a set of competencies. By their nature, they are linked to other TLA 
components such as competency frameworks, assessment rubrics, or talent management systems. This 
requires the establishment of attributes and rules that allow TLA components to process and compare 


18 https://www.medbig.org/working_groups/competencies/index.html 
'9 https://ieeexplore.ieee.org/document/4445693 
0 hittp://sites.ieee.org/sagroups-1484-20-1/ 
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credentials in the same, interoperable way, particularly in Learner Profiles across instantiations of the TLA. 
This also requires a common way to describe, store, and access credentials in order to make fair 
comparisons of the achievement that was performed to acquire them. 


2.4.1, Credential Transparency Description Language (CTDL) 


CTDL”! is a vocabulary comprised of terms that are useful in making assertions about a credential and its 
relationships to other entities. CTDL is modeled as a directed graph using RDF for describing data on the 
web. Like an activity stream, the triple is the basic grammatical construct in making CTDL assertions about 
Things and is comprised of three simple components: a subject, a predicate and an object. This structure 
allows us to make simple statements that enable a rich description of credential-related resources including 
credentialing organizations and specific subclasses of credentials such as Degrees, Certificates, and Badges. 


The comprehensiveness and scope of this specification makes it ideal for defining TLA credentials. As 
shown in Figure 8, two super classes called Agent and Credential define sets of subclasses. The 
primary classes also include the ConditionProfile used to define sets of constraints on the credential 


described by an AssessmentProfile, LearningOpportunityProfile, or 
CompetencyFramework. These are used to express learning goals and outcomes. 


2.4.2 Credential Registry ’ ih 
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information will typically be stored in a Learner Profile. As a 
registry, it does allow users to see what various credentials Figure 8. CDTL Class Structure — Six 
represent in terms of competencies, transfer value, assessment — primary classes structure the data used to 
rigor, third party approval status and more. express learning goals and outcomes. 


Credential Engine developers are using Dublin Core Application Profiles to create systems that 
communicate all aspects of credentials. The Technical Advisory Committee (TAC) promotes collaboration 
across different standardization initiatives that are developing data models, vocabularies, and schemas for 
credentials and competency frameworks. The Credential Registry uses CTDL to capture, link, update, and 
share up-to-date information about credentials so it can be organized and centralized within the registry, 
made searchable by customized applications and linked to from anywhere on the open Web. 


“1 http://credreg.net/ctdl/handbook 
>? https://www.credentialengine.org/credentialregistry 
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2.4.3 IMS Global Open Badges 


Open Badges” are visual representations of verifiable achievements earned by recipients. Open Badges is 
a technical specification and set of associated open source software designed to enable the creation of 
verifiable credentials across a broad spectrum of learning experiences. The Open Badges standard describes 
a method for packaging information about accomplishments, embedding it into portable image files as 
digital badges, and establishing resources for its validation and verification. Open Badges contain detailed 
metadata about achievements such as who earned a badge, who issued it, and what it means. The Open 
Badges 2.0 specification expands on this capability to include versioning, endorsements, and full use of 
JSON-LD. 


Open Badges are expressed as linked data so that badge resources can be connected and reference by other 
systems. This metadata can also be embedded within the graphic. The Open Badges vocabulary defines 
several data classes used to express achievements. There are three core data classes (Assertion, 


BadgeClass, and Profile) that define mandatory and optional properties as well as the restrictions on 
the values those properties may take. Each badge object may have additional properties in the form of an 
Open Badges Extension, a structure that follows a standard format so that other users can understand the 
information added to badges. Assertions are representations of an awarded badge, used to share information 
about a badge belonging to one earner. 


2.4.4 W3C Verifiable Claims 


The mission of the verifiable claims™ is to make expressing and exchanging credentials that have been 
verified by a third party easier and more secure. The basic components of a set of verifiable claims include 
a subject identifier, claims about the subject, claim set metadata, and a digital signature. Both the Entity 
Profile Model and Entity Credential Model consist of a collection of name-value pairs which will be referred 
to as properties in the data model. The link between the two is in the id property. The Entity Profile Model 
defines a subject identifier in the id property, while the claims section of the Entity Credential Model uses 
the id property to refer to that subject identifier. The first working draft of this specification was released 
in August 2017. Additional research is required to better evaluate its applicability to the TLA. However, 
the defined data model and structure of this specification implies that it could connect both semantically 
and syntactically. The data associated with verifiable claims are largely susceptible to privacy violations 
when shared with other TLA components so many of the verifiable claims use cases should be used to 
inform how PII centered on credentials should be protected within the security boundaries of the TLA. 


2.5 LEARNER PROFILE 


There are numerous challenges in creating a lifelong learning profile. What elements of the Learner Profile 
should the learner be in control of? How does learner information pass from one organization to another? 
What authoritative systems have permissions to write to the Learner’s Profile? How do we represent learner 
context? 


Learner Profiles exist in many systems and are extremely diversified in nature. Within the context of the 
future learning ecosystem, it is envisioned that a Learner Profile will house data about people across their 
entire career. Adult learners are characterized by distinctive personal attributes, knowledge, skills and 
competencies that they have gained in different contexts through a process of development and growth. A 


3 https://www.imsglobal.org/activity/digital-credentials-and-badges 
4 https://www.w3.org/TR/verifiable-claims-data-model/ 
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Learner Profile may also include demographic data, student interests, learning preferences, descriptions of 
preferred learning environments, inter/intrapersonal skills, existing competencies and those that need to be 
developed, socio-emotional health, and a myriad of other data. Learner Profiles are dynamic and will 
change over time. As student interests change and they become competent in new areas, the profile will 
update to reflect the latest state of the learner. 


While no formal specifications were used in the 2018 TLA Reference Implementation, numerous insights 
were learned that will influence future research. The 2018 TLA Test and Demonstration exposed a 
multitude of learner-related data was generated and consumed by different TLA components. This 
illustrates the need for a Learner Profile Service that allows different components and activities to exploit 
relevant information about each learner. To maximize the value these components can gain from the learner 
data, a broad and consistent taxonomy is necessary to describe and characterize the different attributes about 
each learner. Business rules are also needed to manage how this data is used across the Future Learning 
Ecosystem. 


Existing Learner Profile specifications (e.g., LIP, PAPI) are somewhat dated. Their bindings are described 
in the XML language which is not appropriate for semantic interoperability. In some ways, these standards 
hinder the process of sharing and reusing Learner Profiles because it is difficult for applications to exploit 
profiles that are stored deep within various servers. Additionally, these specifications do not easily allow 
extensions to the schema with additional information that is relevant to other TLA components. 


2.5.1 Comprehensive Learner Record (CLR) 


IMS Global led the development of the CLR”*, formerly known as the Extended Transcript. It was originally 
designed to support traditional academic programs as well as co-curricular and competency-based 
education by capturing and communicating a learner's achievements in verifiable, digital form. The CLR 
contains data about different learning experiences and achievements including course descriptions, syllabi, 
rubrics, portfolios, performance evaluations, prior learning, internships, and other experiences or 
assessments. As a digital transcript, the CLR closely follows the registrar guidance of the American 
Association of Collegiate Registrars and Admissions Officers (AACRAO) to draw information from an 
institution’s student information systems, learning management systems (LMS), or other internal 
databases.”° The CLR is due to be completed in 2019 but will likely evolve to help foster adoption across 
higher education institutions. Integration with existing institutional reporting systems and data structures 
will be critical in enabling this effort to succeed. The ADL Initiative will continue to monitor this effort to 
measure its applicability to the DoD and other Federal government stakeholders. 


2.5.2 Learner Information Package (LIP) 


The IMS LIP specification’ provides a standard means for recording information about learners. It is 
designed to allow information about learners, including their progress to date and awards received, to be 
transferred between different software applications. In this specification, the learner information is 
separated into eleven main categories of information that re: Identification, 
Qualifications/Certifications/Licenses (QCL), Accessibility, Activity, Goal, Competency, Interest, 
Transcript, Affiliation, Security Key, and Relationship. The LIP specification describes the data structures, 
XML binding, and best practices for the formatting, storage, and exchange of learner information. 


>5 https://www.imsglobal.org/activity/comprehensive-learner-record 
6 https://library.educause.edu/~/media/files/library/2019/1/eli7164.pdf 
77 http://www.imsglobal.org/profiles/index.html 
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The specification supports the exchange of learner information among learning managements systems, 
human resource systems, student information systems, and other systems used in the learning process. LIP 
is an older specification and its reliance on XML impacts the access speed for how these systems can load 
learner information. The future learning ecosystem will require real time learner services that can send and 
update records on the fly. An accessibility for LIP standard”® is extending the LIP Information Model to 
better define accessibility preferences for people with disabilities. This work is intended to benefit all 
learners in situations that require alternative modes of instruction, such as an extremely noisy environment 
where captions are needed for a video or language barriers. 


2.5.3 IEEE Public and Private Information for Learners (PAPI Learner) 


The PAPI Learner specification”? is a multi-part standard that specifies the semantics and syntax for storing, 
retrieving, searching, and exchanging learner information. It defines elements for recording descriptive 
information about knowledge acquisition, skills, abilities, personal information, learner relationships, 
preferences, performance, and similar types of information. An important feature is the logical division, 
separate security, and separate administration of several types of learner information. This specification 
partitions the learner record into six main information types that support extension: 


¢ Learner Contact Information: describes information related to administration. 

¢ Learner Relations Information: stores information about the learner’s relationships to other users 
of learning systems, such as teachers and other learners. 

¢ Learner Security Information: stores information about the learner’s security credentials, e.g. 
passwords, private keys, public keys, biometrics. 

¢ Learner Preference Information: describes preference information intended to improve human 
computer interactions and the automatic adaptation and personalization of systems to specific needs 
of the learner. 

¢ Learner Performance Information: is about the learner’s history, current work, or future 
objectives. PAPI Performance information is primarily created and used by learning technology 
components to provide improved or optimized learning experiences. 

¢ Learner Portfolio Information: is a representative collection of the learner’s works or references 
to them that is intended for presentation and evidencing of his achievements and abilities. 


This standard permits different views of the learner information and addresses issues of privacy and security. 
The data models associated with this specification is not sufficiently complete to cover all the learner data 
that needs to be exchanged between TLA components, particularly those related to 
pedagogical/andragogical aspects such as defining an educational pathway. 


2.5.4 Airmen Learning Record (ALR) 


The vision of the Air Force Learning Services Ecosystem (AFLSE) is to support a continuum of learning 
that deliberately prepares Airmen with the required competencies to meet the challenges of the 21* century. 
The ALR is a comprehensive record of all learning Airmen achieve during their career including 
educational transcripts, training records, performance reports, and ancillary training transcripts. The 
Airmen’s learning record is a centralized location to record all learning, whether it occurs in a specialized 
training or education program, on-the-job, or even off-duty. The Airmen’s learning record will enhance the 


8 https://www.imsglobal.org/accessibility/acclipv1p0/imsacclip_infov1p0.html 
?° http://metadata-standards.org/Document-library/Meeting-reports/SC32WG2/2002-05-Seoul/WG2-SEL- 
042_SC36N0175_papi_learner_core_features.pdf 
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ability to analyze the readiness of the air force by capturing an Airmen's knowledge and skills gained 
throughout the continuum of learning (training, education, and experiences), documenting progress and 
achievements, and identifying gaps and opportunities for growth tied to mission accomplishment from both 
an enterprise perspective as well as an individual level. 


2.6 RECOMMENDATION ENGINE 


A recommendation engine collects data, analyzes the data, filters the data, and makes a recommendation. 
While recommender algorithms are not likely to be standardized, they require computationally accessible 
data to reason over for which technical standards and specifications will play a role. Within the context of 
the TLA, a recommendation engine uses data filtering algorithms that make use of metadata and paradata*” 
about learning resources, competency frameworks and the definitions of competence, Learner Profiles, and 
other data to inform a probabilistic ranking that guides learner interactions. Beyond the existing 
specifications and standards that a recommender must interface with, a recommender must be grounded in 
the science of learning. 


This is substantially different from the recommenders most people are familiar with from their experience 
on Amazon or Netflix. These recommenders use collaborative filtering algorithms, content-based filtering 
algorithms, or a hybrid of the two. These algorithms make recommendations based on other similar user 
choices or on other similar content that someone might be interested in. Within the future learning 
ecosystem, recommendations need to be informed by learning theory. This adds complexity because not all 
people learn the same way which impedes the collaborative filtering approach. Additionally, the process of 
learning is a constantly moving target, so content-based filtering is not an ideal situation either, although 
both will certainly play a role. 


2.6.1 Fast Learning from Unlabeled Episodes for Next-Generation Tailoring (FLUENT) 


FLUENT" is an open source recommendation engine used in the 2018 TLA Reference Implementation. It 
utilized the LRMI metadata attached to various learning resources and information inferred about learners 
to make its recommendations. FLUENT supports the ability to utilize different instructional frameworks 
which provides the ability to deliver content recommendations based on different learning theories. This is 
important as different instructional domains require different instructional strategies. 


2.6.2 Navigator for Integrated Learning Environments (NILE) 


NILE builds upon an existing commercial product that has been historically targeted at the K-12 educational 
domain. They use a Google Maps™ approach to create a recommended learning path for learners to take. 
NILE has a content discovery service that has aggregated millions of educational resources into their 
platform. Recommended learner pathways change based on performance, instructor input, or student 
choices. The ADL Initiative is funding the migration from propriety data structures and interface 
specifications to a TLA-enabled ecosystem to better evaluate how TLA specifications and standards scale. 
NILE includes numerous enabling technologies (e.g., content discovery service) that are also appealing to 
the TLA ecosystem of tools and technologies. 


3° Data about how content was being used that was collected. 
3! https://github.com/soartech/tla/tree/master/learner-profile-interface/src/main/java/com/soartech/fluent 
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2.6.3 Adaptive Instructional Sciences — IEEE (C/LT/AIS) P247.1) 


32 


The purpose of the Adaptive Instructional Systems (AIS) Working Group” is to investigate the market need 
for standards across a group of technologies known as the AIS. The output of the working group will be 
one or more PARs identifying and generating specifications and standards that improve the interoperability 


across adaptive tools and technologies. 
2.7 LAUNCH SERVICE 


The ability to launch learning resources across devices, platforms, operating systems, and browsers is a 
fundamental requirement of the future learning ecosystem. A TLA launch service will ultimately be 
designed as a plugin-based protocol that can launch activities according to different launch standards that 
are applicable to different systems, platforms, or learning modalities. 


2.7.1 xAPI Launch 


The xAPI Launch Specification® is a method of launching xAPI-enabled content by online learning 
modules, static HTML files, offline content, or immersive serious games and simulations. It does not require 
the identity of the learner, the LRS endpoint, or the session information for how the events should be 
grouped. Implementation requires a minimal HTTP request and handles the creation and transmittal of xAPI 
data to the LRS on behalf of the activity. While this specification is mature, it has not been widely adopted. 


2.7.2 cmi5 


The cmi5* Profile replicates many of the features and capabilities associated with SCORM. The 
specification provides definitions, a launch process, a course structure, and runtime communications 
between a learning management system and learning content. The cmi5 allows the setup of several global 
variables including actor, xAPI endpoint and security token, and the automated communication of some 
xAPI statements. The launch portion of cmi5 allows developers to avoid “hard coding” the LRS information 
into the LMS. The cmi5 could be expanded to support a secure, single sign-on experience by decoupling 
the data models and behaviors from the implementation details. Implementation details could be defined as 
part of the cmi5 launch profile which allows it to be used for web and native applications. There are known 
security concerns over the JavaScript credentials and the code is not portable without modification. 


2.8 ASSESSMENT SERVICE 


Assessment within the future learning ecosystem will take many different forms. The nature of assessment 
in a competency-based educational program is more focused on the learner’s ability to demonstrate their 
understanding of key concepts by having them apply their learned skills in different contextual situations. 
The 2018 TLA Reference Implementation used a variety of traditional assessment activities that included 
multiple choice tests/quizzes, Situational Judgement exercises built in Unity3D, mobile applications and 
live group activities. Future implementations require a common approach for communicating competency 
assertions across TLA components and systems. Competency evidence will be aggregated from multiple 
communities inside and outside the organization. Performance indicators, organizational metrics, and other 
systems or databases in use across the enterprise will continually be analyzed and assessed to measure the 
proficiency of individuals and teams within the organization. 


>? http://sites.ieee.org/sagroups-2247-1/ 
33 https://github.com/adInet/xapi-launch 
4 https://github.com/AICC/CMI-5_Spec_Current/blob/quartz/cmi5_spec.md#content_launch 
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Many of the standards and specifications referenced in this document include rubrics for measuring 
performance (HPML, CASE™, Open Badges). Each has its own data structure, evaluation metrics, and 
format and each needs to be considered as part of the TLA assessment service. Controlled vocabularies are 
also required to ensure assessments are all speaking the right language and crediting the right competencies. 


2.8.1 Question & Test Interoperability (QTI®) 


The IMS QTI® specification*®® enables the exchange of item and test content and results data between 
authoring tools, item banks, test construction tools, learning platforms, assessment delivery systems, and 
scoring/analytics engines. The data model is described using UML to facilitate binding to a wide range of 
data-modelling tools and programming languages. The IMS QTI specification has been designed to support 
both interoperability and innovation through the provision of well-defined extension points. These 
extension points can be used to wrap specialized or proprietary data in ways that allow it to be used with 
other test items. QTI 2.2 is very stable and has been built on other stable, adopted versions of QTI. QTI 2.2 
adoption is strong. TLA assessment activities could leverage the data models of QTI to gain interoperability. 


2.9 IDENTITY MANAGEMENT 


The future learning ecosystem requires data exchange across different organizational boundaries and 
inherently different IT enclaves. The 2018 TLA Reference Implementation used Keycloak™, an open 
source single sign-on capability to manage individual access and permissions to all TLA components and 
activities used in the 2018 TLA Test and Demonstration. User names were anonymized through the creation 
of a universal user ID that each component used when communicating data (e.g., activity tracking) about 
each learner. Most tools and technologies are focused on protecting Personally Identifiable Information 
(PIT) inside the organizational firewall; however, there are numerous use cases within the TLA where 
information about a learner needs to be communicated between different organizations. 


This is currently achievable at a minimal level through services like ID.me* and Login.gov*’. ID.me 
provides identify management, authentication and group affiliation verification for numerous government 
and business customers. Login.gov allows users to log into multiple government agencies with a single user 
account. The approaches used to manage these capabilities could be relevant to the TLA. Both platforms 
follow the National Institute of Standards and Technology (NIST) SP 800-63 Digital Identity Guidelines.** 


2.9.1 Privacy and Security for TLA (PS4TLA) 


The PS4TLA* research seeks to make recommendations for implementing a privacy by design model 
where privacy and security strategies are an inherent component of the future learning ecosystem. Areas of 
study include user data characteristics, output characteristics, data location and ownership, data sharing, 
and privacy support mechanisms. This research will result in a requirements specification for tools and 
technologies that manage a learner’s PII and privacy preferences, while also providing individual learners 
with the knowledge required to enact their own privacy control across TLA activities, systems, and services. 


35 https://www.imsglobal.org/question/index.html#version2.2 
36 https://www.id.me/ 

37 https://login.gov/ 

38 https://pages.nist.gov/800-63-3/ 

> https://www.youtube.com/watch?v=opmiFzqfwXo 
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2.10 MISCELLANEOUS 
2.10.1 Learning Tools Interoperability (LTI) 


The IMS LTI Specification*° prescribes a way to connect learning applications and tools with platforms 
like learning management systems, portals and learning object repositories, in a secure and standard manner. 
LTI is comprised of a central core and optional extensions to add optional features and functions. The LTI 
core establishes a secure connection and confirms the tool’s authenticity while the extensions add features 
like the exchange of assignment and grade data between an assessment tool and LMS gradebook. While 
LTI is tool-oriented, the underlying data model includes elements about the learner and could lend itself 
well to a TLA Learner Profile. 


2.10.2 Open Pathways 


Open Pathways"! has the goal to allow publishing and consumption of learning pathways across multiple 
services much like Open Badges. A learning pathway is an organized set of educational goals connected by 
specific digital credentials and a community’s understanding of what people have accomplished in terms 
of competencies, or other measures. This model is published using standardized data formats and a common 
vocabulary across different stakeholder organizations. The Open Pathways effort is in the early stage of 
development and has little community adoption. However, it has potential to define a common taxonomy 
for TLA activities and could inform attributes and rules that TLA learning pathways should follow. 


2.10.3 Sharable Content Object Reference Model (SCORM) 


SCORM defines a specific way of constructing learning content so that it can be shared across learning 
management systems. SCORM is a set of existing standards and specifications that control how content is 
developed, packaged, described, and delivered across systems. A Sharable Content Object (SCO) is the 
most granular piece of training in a SCORM world. The Content Aggregation Model (CAM) determines 
how a piece of content should be delivered in a physical sense. 


At the core of SCORM packaging is a document titled “imsmanifest.xml.” This file contains every piece 
of information required by the LMS to import and launch content without human intervention. The SCORM 
run-time specifies how the content communicates with the LMS while the content is playing. There are two 
major components to this communication. First, the content has to find the LMS. Once the content has 
found it, it can communicate through a series of get and set calls and an associated vocabulary. SCORM is 
entrenched in the traditional distributed learning community and is still widely used for managing online 
learning. It provides an important capability for the TLA in being able to deliver online content. The 
SCORM/xAPI wrapper provides additional capabilities to enable SCORM courses to report to an LRS. 


2.10.4 IEEE Actionable Data Book (ADB) 


ADB is a transformative blend of experiential analytics and rich media, delivered through interactive eBook 
technology. ADB was created as a reference model based solely on open standards. These include the 
International Digital Publishing Forum’s ePub 3 specification, HTML5, Bluetooth, xAPI, and W3C 
packaging standards for interactive content. It appears work on this specification is no longer continuing. 


40 https://www.imsglobal.org/activity/learning-tools-interoperability 


41 https://www.concentricsky.com/articles/detail/open-pathways-connect-badges-to-what-you-care-about 
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2.10.5 ePub 


The ePub specification” defines a distribution and interchange format for digital publications and 
documents. The EPUB format provides a means of representing, packaging and encoding structured and 
semantically enhanced Web content — including HTML, CSS, SVG and other resources — for distribution 
in a single-file container. EPUB has been widely adopted as the format for digital books (eBooks), and with 
new versions continues to increase the format's capabilities to better support a wider range of publication 
requirements, including complex layouts, rich media, interactivity, and global typography features. The 
expectation is that the EPUB format will be utilized for a broad range of content, including books, 
magazines and educational, professional, and scientific publications. 


This specification is relevant to different types of content the TLA can support and was used as part of the 
Personalized eBook for Learning (PeBL). EPUB in general has seen major adoption, but the adoption of 
version 3.1 is inconclusive at this time. Rules are specific and will promote interoperability among tools 
that leverage EPUB 3.1. This specification indirectly competes with the ADB specification. However, ADB 
was designed with consideration of the EPUB specification. 


2.10.6 IEEE P1589 Augmented Reality Learning Experience Models (ARLEM) 


The purpose of the ARLEM specification* is to become an overarching integrated conceptual model that 
describes interactions between the physical world, the user, and digital information to establish the context 
for Augmented Reality (AR) applications used for learning. It will define the required data models, 
modeling languages and their bindings to chosen representation formats (e.g. XML, JSON). The 
specification has not yet been released as a version 1.0 specification. Therefore, a thorough review has not 
been possible. A review of draft documentation found that while ARLEM attempts to formulate a 
standardized taxonomy around Augmented Reality (AR), it doesn’t provide enough vocabulary to support 
Virtual Reality (VR) or Mixed Reality (MR) learning environments and therefore requires extrapolation. 
As this specification matures, the ADL Initiative will continue to investigate to determine how these 
learning activities can integrate with the TLA. 


* https://www.w3.org/Submission/2017/SUBM-epub31-20170125/ 
43 http://arlem.cct.brookes.ac.uk/ 
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3. 2018 TLA REFERENCE IMPLEMENTATION 


This section describes the logical, physical, and operational implementation of the FY18 TLA 
demonstration. It introduces the interface definitions that will form the core of future TLA implementations, 
as well as full descriptions of the commercial items used to provide the current test implementation. The 
document is organized like the architectural elements defined in a system/subsystem design document 
(SSDD), albeit the requirements traceability section is tailored out. A research summary and lessons 
learned, from a technology standpoint, is added as Section 5. The research effort did not maintain detailed 
requirements, although high level functional requirements for the TLA implementation were collected and 
defined at a series of Technical Integration and Planning (TIP) meetings held throughout the year leading 
up to the 2018 TLA Test and Demonstration held at Fort Bragg in August of 2018. 


3.1 LOGICAL ARCHITECTURE 


The TLA Logical Architecture decomposes complexity by separating the TLA into a set of related technical 
services and principles that support the operation of the system. The service layer in Figure I acts as the 
bridge between TLA components and shared data stores. Each service exposes the stored data to an 
application so that information can be transformed into other meaningful data used by other TLA 
components. Each service includes control logic and user interfaces for a set of functions as depicted in 
Figure 9. The data contracts between data and service layers are shown based on the nature of the data 
exchanged. The behavior and functionality of each service is defined and aligned with a TLA business 
function. Input-output data flows are identified and aligned with the required TLA data stores. Data models 
and protocols are defined around candidate TLA standards and specifications. 


Specific implementation of data interfaces is described in Section 3.3. The service layers include: 


e User Management: Handles protection of Personally Identifiable Information (PID), login 
credentials and identification. In the 2018 implementation this was done primarily by Keycloak*. 
Keycloak™ is an open source Identify and Access Management solution. This service will typically 
be inherited from the parent organization implementing a TLA instance. 


e Resource Management: For the 2018 event, the physical location and endpoints of TLA 
components were managed manually via a static index. Resource Management services are already 
used by the ADL Initiative stakeholders to manage dorm rooms, live training logistics, simulator 
schedules, and other capabilities. This data could be used to inform the TLA. 


e Competency Management: Tracks overall competency state for selected goals and makes 
assertions based on evidence provided via xAPI. The Competency and Skills System (CaSS)* 
performed the competency service for the 2018 event. The CaSS project includes a Learner Profile 
as an internal data store that archives mastery estimates and user metadata for each learner. 


e Activity Management: Manages the thread of assigning content to satisfy an identified goal. 
Activity management in 2018 included a recommender service that prioritized recommendations 
based on inferences made from the learners’ performance. The FLUENT system performed the 
activity management functions in the 2018 TLA Reference Implementation. FLUENT also acted 
as the primary student interface for the TLA. 


“4 https://www.keycloak.org/index.html 
45 
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e Content Management: Manages the registration of content. In the 2018 Implementation, this was 
a Static file called the Activity Index, which included metadata for all content in the 2018 content 
database. It was managed in an offline manual process, and physically co-located with the FLUENT 
instance. 


e Decision Management: Manages the presentation and analysis of aggregate data for making 
learner, curricular, facility and other decisions. In the 2018 TLA Test and Demonstration, this was 
an offline process using commercial tools built into the LRS, or with exports in Microsoft Excel. 


Within the logical services architecture, learning content is technically not part of the TLA, but the content 
was stored on TLA controlled servers for the purposes of the 2018 TLA Test and Demonstration, so it is 
included in this description. Content viewers establish data contracts with the TLA ecosystem by acting as 
“boundary objects.” The Content viewers and a Modified version of Moodle™ thus act as the Learning 
Record Providers (LRP), generating xAPI statements of user learning experiences, and managing content 
launch and user ID tokens. The LRPs generate xAPI statements that are streamed to the User LRS. These 
statements are the raw evidence that the CaSS uses to make assertions of competence. 


These assertions are transformed into estimated levels of competency (mastery estimates) for each learner 
in the system. A recommendation engine (FLUENT) uses competency information and Learner Profiles 
schedule the delivery of activities through a series of prioritized recommendations. The ADL Initiative 
managed services include the LRS, authentication, identity management, content discovery (resource 
management), and launch services. 


3.2 PHYSICAL ARCHITECTURE 


The 2018 TLA Reference Implementation was installed on a set of seven virtual machines hosted by AWS. 
AWS provides the backend platform hosting, virtualization, and Domain Name Service (DNS) resolution 
functions. Six of the servers were procured under contract to USALearning, maintained by the ADL 
Initiative; the seventh was separately procured by the vendor on AWS under the prime CaSS contract. The 
specifications for each server are listed in Table 4. All seven servers used Ubuntu 16 LINUX as the 
operating system (OS). Web server configuration varied by vendor, with some teams utilizing traditional 
deployment and others using containerization frameworks. The Server instances communicate between 
themselves using HTTP over TCP/IP internally to the Amazon campus. The application ports and protocols 
used to access each service are listed in Table 5 and described in Section 3.5. 
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Figure 9. Logical TLA Services Architecture — The TLA is implemented as a combination of multiple services (and microservices) and their interactions. Hence 
the communications between services and their coordination is vital for a successful realization of the architecture. Business functions within the TLA are often 
comprised of multiple services. Therefore, services cannot always be mapped directly to a specific TLA system or component. 
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Table 4. AWS Server Specifications — The 20/8 TLA Reference Implementation utilized a set of 7 virtual machines 
hosted by AWS. CaSS, FLUENT, Moodle™, and two instances of the Learning Locker LRS were set up to run with a 
content server and another server configured to run essential TLA services such as single sign-on authentication, 
launch, and discovery. 


Snap 


D ipti EC2 T Disk Vol T Si 
escription ype isk Volume Type orage Throughput storage 


VM1-LRS General Purpose SSD (gp2) 
VM2-LogLRS General Purpose SSD (gp2) 150 


VM3- 
AD lSarver General Purpose SSD (gp2) 100 


VM4- 

SoarTech X-Large | General Purpose SSD (gp2) 150 
ee Medium | General Purpose SSD (gp2) 200 150 
Content P gP 


VM6- 
VM7-CaSS General Purpose SSD (gp2) 150 


While services and systems could be consolidated to run on fewer servers, the simplicity of this approach 
reduced risk and delineated clear boundaries for what will run on each system in the cluster. Moodle™ 
requires significant computational resources for each connected learner. Each LRS and the FLUENT 
recommendation engine also require significant resources for storing the different data elements about 
systems, learners, and content. A second LRS was added to support the logging of system components 
about different ongoing activities using xAPI (e.g., data about recommendations, competency assertions, 
learner inference, etc.). The six elastic IP addresses and 1000GB of data transfer per month was also added 
to AWS hosting requirements. 


As shown below in Figure 10, the Student and Instructor Workstations communicated with the AWS 
servers over the commercial internet using the Hypertext Transfer Protocol (HTTP), accessed through a 
web browser. The Google Chrome™ 64-bit browser, version 69.0.3497.100, loaded on top of Windows 10 
OS was used as the principal interface. The initial login page and credential management were provided by 
a Keycloak™ Server. Keycloak™ is a commercial, open source, single sign on application that also 
generates 28 key user tokens to protect PII by avoiding the passage of names, SSN, or other private data. 


The FLUENT user interface (UI), hosted on the SOAR Technologies server, provided the main control 
panel page resources. Learning content pages used a set of browser-based content viewers to interact with 
learning content. A launcher application installed on the workstations, as well as the Moodle™ Server, 
managed launch of the individual content viewers of the selected type. A Keycloak™ client application 
installed on the workstations centrally manages logins. All content viewers except PEBL could launch in 
new browser sessions on the student and instructor PC workstations. The Personal eBook Learning (PEBL) 
application launched from an iPad™ personal data tablet device, using the user login to the Keycloak™ 
app to match user tokens to identify the correct device to launch. This overall physical architecture is 
depicted in Figure 3 in the first section of this document. 
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Figure 10. 2018 TLA Reference Implementation Physical Architecture — The TLA’s physical architecture defines 
the system elements that can perform the functions of the logical architecture. The 2018 TLA Reference 
Implementation is built from systems, system elements, and all necessary physical interfaces between these elements. 


3.3 SYSTEM COMPONENTS 
3.3.1 Activity Registry 


An Activity Registry is an approach to capturing, connecting and sharing data about learning resources 
available to an organization. Key features include the ability to generate and manage metadata, taxonomies 
and ontologies, the alignment of content with competencies, paradata, semantic search services, and 
machine-actionable metadata. In the 2018 TLA Reference Implementation, the Activity Registry combined 
cataloging information, usage, assertions, and analytical data into a single, sharable metadata repository. 
The Activity Registry has a trusted relationship with connected learning resources and other TLA services 
like launch and discovery. This growing collection of data about learning resources can be consumed by 
any TLA component. The Activity Registry was deployed as part of the FLUENT recommender; however, 
future versions will be implemented as a stand-alone capability. 


A JSON metadata file was generated for each activity. Metadata was created following the LRMI 
specification and made specific use of the extension for educational alignments. Beyond the general 
metadata overhead (e.g., publisher, name, location), a very limited set of differentiators were implemented 
for 2018. The FLUENT recommender required 5 different metadata types for making its recommendations. 
Metadata was a semi-automated process using a spreadsheet to manually enter all metadata fields. A script 
formatted all fields into LRMI formatted JSON file that was uploaded to the Activity Index. 
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3.3.2 xAPI Activity Streams 


Each LRP is uniquely situated to produce learning records that connect a user to learning experiences 
within an activity. The LRP is responsible for the formatting and population of a learning record that meets 
xAPI requirements. These learning records can then be transmitted to the LRS. Within the 2018 TLA 
Reference Implementation, the xAPI was used exclusively to track individual learner experiences across 
different learning activities. System logs were also formatted into xAPI and sent to a separate LRS 


3.3.3 Learner Record Store 


The xAPI-enabled learning activities generate statements, or records of e-learning which include a basic 
“triple” structure consisting of actor, verb, and object. These statements are transmitted over HTTP or 
HTTPS to an LRS. The xAPI requires a central storage capability called a Learner Record Store (LRS). An 
LRS is a data store that serves as a repository for learning records collected from connected systems where 
learning activities are conducted. It is an essential component in the process flow for using the xAPI. The 
2018 TLA Reference Implementation used the open source Learning Locker LRS*°. Its main function is to 
store and retrieve the data that's generated from xAPI statements such that all other TLA components may 
access it without being dependent on direct communication with each other. With the statement structure 
of the LRS records, native Learning Locker dashboards were generated using a number of different actor, 
verb, and object combinations. 


3.3.4 Competency Management System 


A competency management system manages evidence of an individual skills, knowledge, abilities, 
attributes, experiences, personality traits and motivators to predict their value towards effective 
performance in a job. Competencies might include technical skills, business skills, leadership skills, people 
skills, ethics, professionalism, or any number of topics related to a job. Within the context of the TLA, the 
Activity Stream(s) stored in the LRS provide the evidence upon which competency assertions are made. 
Key to establishing assertions of competency from an Activity Stream, is the reconciliation between the 
Actor ina statement (stored in an LRS) and a persistent Learner Profile. As shown in Figure 11, the 2018 
TLA Reference Implementation utilized the open source CaSS*’ to manage this evidence and infer 
competency against the competency framework. 


For the 2018 TLA Reference Implementation, CaSS pulled xAPI statements from the LRS and checked the 
verb to determine whether the statement claims any evidence of competency. For statements that infer 


competency, the ObjectID from the xAPI statement was compared to the Activity Registry to determine 
which competencies a statement is providing evidence for. The primary output from CaSS is a series of 
mastery estimates for an individual over time. This data is managed within a service layer that provides a 
common, semantic, and syntactically accessible network service for other systems to leverage. CaSS also 
enables assertions about an individual's competencies to be collected, processed, and incorporated into a 
machine-readable Learner Profile. Within the 2018 TLA Reference Implementation, AMQP was used to 
push this information to other TLA components and other TLA components were able to write to the 
Learner Profile by making assertions through CaSS. 


46 https://www.ht2labs.com/learning-locker/ 
47 www.cassproject.org 
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2018 TLA Competency Management Process 


FLUENT: FLUENT uses data from the CaSS Learner Profile (retrieved via an API) to recommend the next activity to the learner via its 
algorithms. FLUENT algorithms also use the learner’s sequence of activities and results (from the LRS), plus the learner’s traits and 


preferences (stored in the CaSS Learner Profile as assertions about purpose-defined frameworks). FLUENT continually updates its views of 
learner traits and transmit them to CaSS as assertions. 
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Figure 11. Competency and Skills System — The 2018 TLA Reference Implementation used CaSS to create, manage, 
and share information about competencies with other TLA components. 


3.3.5 Learner Profile 


In the 2018 TLA Reference Implementation, the Learner Profile was developed as part of the CaSS project. 
It primarily contains information about competencies, competency assertions, and credentials. Current and 
future ADL Initiative research is attempting to understand how different learner characteristics influence 
the intelligent delivery of content to optimize how a person learns. For example, the 2018 TLA 
test/demonstration populated the Learner Profile with additional learner inference data from FLUENT that 
helped the system rank and prioritize recommendations tailored to each learner. Future implementations 
will continue to explore how a Learner Profile might influence other TLA components. 


3.3.6 Recommendation Engine 


The open source FLUENT recommendation engine uses information about competencies, content, and 
learners to recommend different learning activities for a user to follow at a given point in time. For the 2018 
demonstration, the competencies were divided into knowledge domains which were linear in nature and ill- 
defined tasks which required a higher level of understanding to apply the learned knowledge and skills. 
FLUENT also made use of a small subset of learner information that primarily included mastery estimates 
for each competency. 


To make data-driven recommendations, FLUENT processes information about a learner including their 
history and context for learning. Learner inference data was derived from the Learner Profile, the LRS, and 
paradata from individual learners. FLUENT’s Collaborative Filtering algorithms look at a user’s past 
behavior and aligns that with the behavior of other users to make a recommendation. Any information 
derived about a learner was written back to the CaSS Learner Profile using xAPI assertions. FLUENT’s 
content-based filtering algorithms utilize a series of discrete characteristics about an activity, as described 
in its metadata, to recommend additional items with similar properties. 
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3.3.7 Single Sign On 


Single sign-on (SSO) is a property of the TLA’s access control of multiple related, yet independent, 
software systems. With this property, a user logs in with a single ID and password to gain access to all TLA 
connected systems. While this is not a formal component of the TLA, the ADL Initiative expects that any 
TLA component will need to operate within an ADL Initiative stakeholder SSO environment. Within the 
2018 TLA Reference Implementation, Keycloak™ was used for this purpose. This capability has been 
integrated to protect Personally Identifiable Information (PID) and to provide information security for 
learners, content, and organizations. 


3.4 CONCEPT OF EXECUTION 


The basic user thread of execution in the TLA demonstration system starts with the Keycloak™ LOGIN 
and proceeds otherwise through the FLUENT UL. Learners select goals, and then select from prioritized or 
un-prioritized content work towards that goal, launch the content, and closeout their performance. The 
experience closeouts, and any related assessments, are recorded as xAPI statements in the LRS. The goal — 
activity — content thread continues in a loop until the goals are all satisfied. 


Two micro-services were used to communicate data between the different systems. A JSON/REST API 
was used to provide synchronous services between public facing components such as the recommender UI 
and the LRS. A RabbitMQ-based implementation of AMQP was used to provide immediate data transfer 
between server-side components, such as the mastery estimates coming from CaSS which needed to be 
communicated to FLUENT. Within the 2018TLA Reference Implementation, all data exchange APIs were 
required to support CRUD (Create, Read, Update, Delete) Operations with error messaging. System logs 
were embedded into xAPI statements and sent to a dedicated logging LRS, providing additional insight into 
the behaviors of each component. 


The 2018 Test and Demonstration showed that a pub/sub architecture is necessary for achieving the desired 
performance for the TLA. The quantity of messages and data that will be present when deploying at scale 
is very large. While AMQP demonstrated the effect push can have on end users at a small scale, it may not 
be adequate from a scalability standpoint. FLUENT reported latency issues with the use of linked data and 
identified it as a performance bottleneck when attempting to achieve real-time data sharing performance. 
Retrieving data involves following links many layers deep to retrieve sub-parts of the necessary data. If the 
linked data structure is complex, the number of interface calls required to retrieve the whole data structure 
can number in the hundreds or thousands. Other work-arounds need to be investigated. 


The basic flow for the Run Time Mode is shown in the Unified Modeling Language (UML) sequence 
diagram of Figure 12. There are administrative and diagnostic modes using the native User Interfaces of 
Moodle™, CaSS, Keycloak™, and Learning Locker which are not shown in the diagram as part of the 
overall flow of execution. These interfaces were not intended to be accessed by learners. Administrative 
and diagnostic modes were used to: 


e Modify the content discovery and metadata in the activity index. 

e Unpin live-locked content by forcing a state change in Moodle™. 
e Unpin live-locked competencies by forcing a state change in CaSS. 
e Forcing global sign outs if Keycloak™ had an error. 


Orange boxes represent system components in the data layer, blue boxes show components in the service 
layer, and green boxes represent the boundary object of the ecosystem, as depicted in Figure 9. 
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2018 TLA Test and Demonstration. The learner signs into the TLA via Keycloak™ and is then directed to the FLUENT User Interface where they remain in an 


Figure 12. TLA Reference Implementation Concept of Execution (COE) — This describes the overall learner experience from their first-time logging into the 
infinite loop of receiving content recommendations until all competencies are completed. 
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3.5 INTERFACE DESIGN 


The TLA is comprised of a collection of open specifications that define the specific message structures 
supported by the application programming interfaces used to expose TLA data, services, and systems. The 
next sections describe the interfaces used in the 2018 TLA Reference Implementation. These interfaces 
represent the operationalization of service contracts defined in the logical architecture. 


3.5.1 Interface Identification and Diagrams 


Figure 13 shows the specific data interfaces between the low-level components of the System configuration 
items in the TLA Reference Implementation (which are shown along the top as swim lanes in the diagram. 
The FLUENT User Interface is the principle interface for the learner. It has the following functions: 


e FLUENT launches via HTTP when the user logs into Keycloak™ successfully. 

e FLUENT connects via REST calls to the recommender service within FLUENT to obtain the 
current state of competencies and set user selected goals, 

e FLUENT sends launch requests to the launch service via AMQP, connects to Keycloak™ via 
AMQP, and launches the content viewers as second browser sessions. 

e FLUENT connects to the CaSS using HTTP, shares assertions using AMQP and generates xAPI 
statements to the system LRS by sending JSON over AMQP. 


The TLA launch services were embedded into FLUENT which provided the following capabilities: 


e = ©The user can launch content for static content viewer or xAPI video viewer via HTTP connected to 
launch client. This browser sends xAPI messages to the user LRS via JSON over AMQP. 

e SCORM and assessment content launch in Moodle™, which loads in its own browser session with 
the launch function natively communicating through HTTP and generating xAPI statements to the 
user LRS in AMQP over HTTP. 

e PEBL and PALMS content launches in their own browser session and generate xAPI to the user 
LRS natively using AMQP over HTTP, they host the launch function natively. 


3.5.2 Advanced Message Queuing Protocol (AMQP) 


AMQP is an open standard application layer protocol for message-oriented middleware to enable acommon 
messaging service between TLA components. It provides an extensible markup language (XML) based 
middleware application layer protocol for exchanging information via HTTP requests. 


3.5.3 Representational State Transition (RESTful) 


REST is an architectural paradigm that defines a set of constraints to be used for creating web services, 
which maintain data and control logic on the edges of communication and uses a series of HTTP based 
requests to a Universal Resource Locator (URL) or specified address for a network resource 


3.5.4 Experience Application Program Interface (xAPI) 


The xAPI is a specification currently going through the IEEE-LTSC standards development process 
(https://www.tagxapi.org/). The main components of xAPI are the data structure called statements and the 


data storage/retrieval capability called the Learning Record Store (LRS). The xAPI specification has 
stringent requirements on the structure of this data and the capabilities of the LRS. Statements are data 
triples that use an actor, a verb, and an object to describe any experience. Each statement also includes 
timestamps and unique, resolvable identifiers. The transport is HTTP/HTTPS with JSON payloads. 
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Figure 13. TLA Reference Implementation Components and Interfaces — Jn its current form, the TLA follows a modular open systems architecture that 
leverages technical standards to support a loosely coupled and highly cohesive system structure. This allows TLA components to be added, modified, replaced, 
removed or supported by different vendors throughout their lifecycle. TLA components use carefully defined execution boundaries layered onto an integrated 
framework of shared applications, services, and data. TLA standards are published documents that establish key interface specifications, communication protocols, 
and data structures designed to facilitate interoperability between TLA components, enable compatibility with a wide range of learning content, and maximize the 
efficiency for how people learn throughout their career from “hire to retire.”’ These standards form the fundamental building blocks for life-long learning by 
establishing consistent protocols that can be universally understood and adopted by any TLA component to enable data exchange about learners, activities, and 


experiences. 
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3.6 COMPUTATIONAL RESOURCES 


The ADL Initiative performed a load test of the server configuration listed in Table 5 to determine the top 
end on the performance of the current TLA Reference Implementation. This test was run over the course 
of 90 minutes and simulated 1,000 instances of a single user signing in, viewing a course, opening and 
closing a SCORM course, taking a quiz, viewing all users enrolled in a course, and then signing out. The 
system failed at 10,000 users with Moodle™ unable to handle the load, failing on CPU overload. Memory 
is not provided as it did not represent a significant resource limit in any case. The network uplink limited 
component was the ADL Initiative server, likely Keycloak™ request for user ID. 


Table 5. TLA Reference Implementation Application Layer Protocol Port Usage — Just as the IP address 
identifies the computer on a network, the network port identifies the application or service running on the computer. 
This allows TLA servers in the testing environment to run multiple applications and/or services. 


HTTP IP Address / URL Endpoint 
Service 


CaSS cass.tla.cassproject.org http://cass.tla.cassproject.org/api 

Moodle™ moodletla.usalearning.net 80 http://moodletla.usalearning.net 

Static content contenttla.usalearning.net 8000 http://contenttla.usalearning.net:8000 

viewer 

Video player contenttla.usalearning.net 8001 http://contenttla.usalearning.net:8001 

User LRS Irstla.usalearning.net 8001 http://rstla.usalearning.net:8001/data/xapi 

Assessment contenttla.usalearning.net 8002 http://contenttla.usalearning.net:8002 

Static content contenttla.usalearning.net 8003 http://contenttla.usalearning.net:8003 

FLUENT UI soartla.usalearning.net 8004 http://soartla.usalearning.net:8004 

PALMS contenttla.usalearning.net 8004 http://contenttla.usalearning.net:8004 

Keycloak™ aditla.usalearning.net 8081 http://aditla.usalearning.net:808 l/auth 

Launch client adltla.usalearning.net 8081 http://aditla.usalearning.net:808 l/application- 
launcher/client 

Launch aditla.usalearning.net 8081 http://adltla.usalearning.net:808 l/application- 
launcher/rest 

Discovery adltla.usalearning.net 8085 http://adltla.usalearning.net:8085/endpoints 

FLUENT UI soartla.usalearning.net 8778 http://soartla.usalearning.net:8778 

support 

FLUENT soartla.usalearning.net 8979 http://soartla.usalearning.net:8979 

Activity soartla.usalearning.net 8989 http://soartla.usalearning.net:8989/activity- 

Index index/activities 

Learner soartla.usalearning.net 8999 http://soartla.usalearning.net:8999 

Inference 

System LRS logtla.usalearning.net 80 http:/Aogtla.usalearning.net/data/xapi 


The point to point nature of all messaging within the 2018 TLA Reference Implementation was also a 
limiting factor in our ability to aggregate real-time data from different systems into a TLA-driven 
dashboard. The only analytics available within the 2018 implementation were those that were resident 
within the Learning Locker LRS. This limited the analytics to xAPI statements which did drive interesting 
statistics about content usage. However, the 2019 migration to a data streaming architecture enables a data 
strategy that utilizes data from multiple sources to drive analytics and visualizations. 
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3.7 LESSONS LEARNED AND FUTURE RESEARCH 


The use of the system LRS to record assertions and inferences generated almost 75% of the xAPI 
statements. This is a performance bottleneck for the system as it scales. Since the assertions and inferences 
are developed continuously, this represents system data with a relatively high Quality of Service (QoS) 
requirement. This, along with several other concerns drives a desire to move to a streaming architecture for 
FY19, to ensure scalability and QoS. 


The FLUENT application basically served as a singleton for the entire TLA 2018 Reference 
Implementation because it provides the primary user interface and manages the state space of the execution. 
The primary interface for the TLA will likely migrate to a common TLA portal. Moreover, a true SOA 
paradigm should be stateless between service boundaries. Moreover, the FLUENT application was single 
threaded and caused some performance delays during test and execution. Moving the recommender service 
to the system edge and improving its performance are goals for FY19 follow on work. 


The CaSS calculations of convergence on competency is very sensitive to the weights assigned for the 
contribution of each content type for making assertions. These weights were determined without scientific 
rigor, and the FY19 architecture must move them to the edge to allow for human in the loop intervention. 
Moreover, some students ended up reaching an asymptote below the arbitrary 80% threshold set for 
competence and it was impossible to push the system farther forward. 


The FLUENT recommender relied largely on the metadata assigned in twelve categories to select 
recommendations. It did not address learning path recommendations, as goals were selected manually by 
users. The FLUENT UI presentation of the CaSS maintained competency tree did not provide enough 
information for users to optimize their own learning path. This automation “surprise” should be addressed 
by the FY19 effort. 


Keycloak™ was not implemented correctly, and while it was only chosen as one solution among many 
possible for SSO, the interface between the TLA services and the backend services used to protect PII and 
otherwise maintain information assurance should be addressed to facilitate future transition efforts. 


The Static content viewer provides a very simple implementation for recording experiences with electronic 
publications. PEBL content required a full copy of the PEBL content repository and specially rigged content 
on each client. The ADL Initiative is reviewing the electronic publication technology strategy for FY19. 


The Moodle™ version used was not the USALearning version, but required modifications for 
interoperability with Keycloak™, xAPI, and launch client. The CODIAC SCORM course did not allow 
fast forwarding to the appropriate point for the specific goal pursued and required the students to complete 
the entire course. The LMS migration and integration plan must be reviewed for FY19. 
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4. 2018 TLA TEST AND DEMONSTRATION 


The ADL Initiative worked with the Institute for Defense Analysis (IDA) and the Special Warfare 
Education Group (Airborne) -SWEG(A), to plan logistics, scope, scale, and domain for the 2018 TLA Test 
and Demonstration. IDA designed the experiment using SWEG(A) students and the ADL Initiative 
integrated TLA systems and developed the course and its content. The ADL Initiative staff was augmented 
by vendors under contract to work on various TLA-related systems. 


During the initial meeting with SWEG (A) representatives, a request was made to move away from the 
cyber and IT training used in the 2017 demonstration. They requested instruction that was more relevant to 
their mission. In order of preference, they preferred small unit training, 18D medical training, or 18E 
communications / radio training. The USJFCOM Combat Observation and Decision-Making in Irregular 
and Ambiguous Conflicts (CODIAC) course was ultimately approved as the foundation for the 2018 TLA 
Test and Demonstration. To assess the TLA, several research questions were posed. The 2018 TLA Test 
and Demonstration had the following research objectives: 


e Assess the TLA functional requirements, does the TLA 2018 Reference Implementation 
demonstrate the intended functionality described below categorically? 


e Assess the conformance to candidate specifications to allow TLA components and activities to 
communicate with each other. 


e Assess successfully indexing, discovering (service discovery within the architecture), and 
launching a wide variety of learning activities across devices, operating systems, and modalities. 


e Assess the ability of the TLA to track learner competence within a competency framework. 


e Assess the ability to produce user learning activity recommendations based on the current state of 
learner competence. 


e Evaluate the metadata strategies used within the 2018 TLA Reference Implementation. 


e Assess the potential of the TLA for supporting cross-activity dashboards for instructors, learners, 
administrators, and others. 


e Assess the TLA reference implementation for system-ilities (reliability, availability, 
maintainability, scalability, interoperability, portability, or others). 


e Assess usability and user experience when interacting with the 2018 TLA Reference 
Implementation. 


e Evaluate usability of the 2018 TLA Reference Implementation of the TLA by populations other 
than end users. 


e Evaluate the technical viability of implementation and the diffusion potential compared to previous 
versions. 


e Document system usage to determine whether the system functions as expected. 
e Identify costs and potential benefits to different communities. 


e Evaluate learning potential and the ability to enhance the learning experience. 
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A mixed-methods approach incorporating Delphi techniques, surveys, focus groups and a quasi-experiment 
was used for assessing the TLA. For assessing technical decisions, documentation quality, and diffusion 
potential, Delphi methods were used with panelists reacting to the API specifications, use-cases, and 
implementation aids. To assess the APIs, a testing implementation of the TLA was deployed and the 
functional and non-functional requirements tested with human subjects. To assess the functionality and 
non-functional requirements of the 2018 TLA Test and Demonstration system, the unit of analysis for 
examining behavior and results can no longer reside with human subjects but moves to the behavior of the 
components and APIs comprising the reference implementation and the results as measured by the integrity 
of the system’s behavior compared to requirements. These levels will be assessed through the usage of 2018 
TLA Test and Demonstration system level data to determine descriptively the behavior of the testing 
implementation, its potential, and experiences and levels of effort required to develop for and integrate with 
the test and demonstration system using the TLA architecture. 


4.1 TECHNICAL PLANNING 


Technical planning and integration meetings were held throughout the year in preparation for the overall 
2018 TLA Test and Demonstration event. The first technical interchange meeting was held over 2 days in 
early March. The purpose was to bring TLA component developers, activity providers, researchers, and 
end-users together to plan logistics, scope, scale, and domain for the TLA demonstration at Fort Bragg in 
August 2018. The attendance of these events by representatives from the Special Warfare Education Group 
(Airborne) — SWEG(A), at Fort Bragg provided the opportunity for dialogue about IT requirements, target 
audience, instructional content, and other operating constraints. Other attendees included the ADL Initiative 
SETA staff, government personnel, the Institute for Defense Analysis (IDA), and vendors that are working 
on different elements of the TLA and the 2018 demonstration at Fort Bragg. 


As shown in Table 6, numerous planning meetings were held. The first meeting focused on providing 
background information to build a common understanding of all relevant TLA components, activities, and 
the CODIAC Program of Instruction. Attendees were divided into three working groups based on their 
expertise. Risks were identified within each working group. Mitigation strategies were discussed, progress 
was updated, and a schedule of activities and dependencies was created and agreed upon by all participants 
within each working group. The three working groups included: 


1. CODIAC Content Demonstration 
e Decompose the 200-hour CODIAC course into a smaller subset that can be used to support 
the 2018 TLA Test and Demonstration. 
e Create a related competency framework. 
e §©Align/augment existing course content. 
e Decompose and realign existing CODIAC source materials, identify new related content, 
design new activities. 
2. 2018 TLA Technical Integration 
e Technical development and integration of TLA components and services. 
e Protection of PII and IA conformance. 
3. CODIAC Content Instrumentation (This working group was eventually absorbed by the Content 
Demonstration working group) 
e Define methods for instrumenting CODIAC learning activities for optimized reporting, 
analytics, and discovery by the TLA recommendation engine (FLUENT). 
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To help integrate the products from each working group, regular meetings provided the project team with 
an understanding of the technical underpinnings developed in each working group. A collaborative 
SharePoint site was established to help the team maintain a clear understanding of critical dates, milestones, 
objectives, and next steps for successfully executing the 2018 TLA Test and Demonstration in August 2018. 
A pilot test was scheduled for late July where a substantially completed walk-through of the course was 
performed. Numerous issues arose, and a second pilot test was more successful in early August. The ADL 
Initiative and IDA worked with SWEG(A) to handle all logistics including IRB approvals, IT setup and 
deployment, participant management and administrative tasks. 


Table 6. Key Planning Events - Numerous in-person team meetings were held throughout the year. Working groups 


met weekly. 
Month Key Planning Event Accomplishments 
February Initial Planning Meeting The ADL Initiative and SWEG(A) staff meet at Fort Bragg to discuss 
all aspects of the 2018 TLA Test and Demonstration 
March 6-7 Technical Integration and In person planning meeting focused on engineering efforts for the 
Planning Meeting #1 2018 TLA Reference Implementation, ISD efforts for the 
curriculum/course development, and implementation guidance for 
associated activities 
May 7-8 Technical Integration and In person planning meeting focused on engineering efforts for the 
Planning Meeting #2 2018 TLA Reference Implementation, ISD efforts for the 
curriculum/course development, and implementation guidance for 
associated activities 
July 21-24 Pilot Test In person testing of the 2018 TLA Reference Implementation with 
representative sample of learning resources connected and 
substantially complete 
August 13-17 | TLA Test and One-week course provided to SWEG(A) students. Pretest and 
Demonstration surveys completed by students prior to the course. A post-test, 
surveys, and focus groups were completed by students on the last 
day of the course. 


4.2 TLA CONTENT DEMONSTRATION 


A key emphasis was placed upon the content demonstration aspects of this event in addition to the 
instructional relevance of the 2018 TLA Test and Demonstration event. From this perspective, instructional 
content had to be presented in a logical and intuitive way that maximized our ability collect, analyze and 
display learning data to the relevant target audience (student, instructor, administrator). By instrumenting a 
wide array of related content, the demonstration was expected to generate a large amount of inter-connected 
human performance data (based on individual performance) that can be used to demonstrate the potential 
of the TLA for observing learner progress across learning modalities. 
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The TLA is content agnostic; it should be able to 
equally support any-and-all subject areas. However, to é 

enable empirical testing of the _ reference o.6..) 11 o 
implementation, the Combat Profiling domain as taught ERA QE TTY AND DOE AAA PS RY TRAISCN AR ARC AMBRE SN ESL 
in the CODIAC course shown in Figure 14 was 
selected. Major work efforts included the 
decomposition of an existing course into approximately 
14 hours of instruction, building a related competency 
framework, establishing levels of mastery, and the 
alignment of learning resources with specific 
competency objects. The intention of the working group 
was to expand the aperture for the types of content the 
TLA can utilize by going beyond traditional DL content 


and making use of serious games and simulations, live 


Figure 14. CODIAC — Inspired by the US Marine 
Corps’ Combat Hunter program, CODIAC was 
participants may encounter in their lifetime. used as a foundation for the 2018 TLA Test and 


exercises, eBooks, and other adaptive content that 


Demonstration. 

4.2.1 CODIAC Course Description 

The CODIAC Program of Instruction (POI) includes systematic training designed to improve cognitive 
skills, showing personnel how to read the human terrain, establish a baseline, detect an anomaly, and make 
decisions “left of bang.” The CODIAC course is presented as a 200-hour course that includes nine modules 
with the last module consisting primarily of scenario-based exercises. Of the eight remaining modules, each 
has explicitly defined Enabling and Terminal Learning Objectives (ELOs/TLOs). Multiple courses of action 
were discussed by the working group about how to structure the course for the 2018 TLA Test and 
Demonstration. Initial thoughts were to find content that skimmed across all CODIAC modules to lightly 
provide an understanding of the entire CODIAC POI. However, after further reviews and discussion, the 
decision was made to provide a more robust curriculum around the entire course. 


The 2018 TLA Test and Demonstration focus on delivering instructional content and learning activities that 
support a competency-based approach to delivering the CODIAC program of instruction. This course has 
well-defined learning objectives, instructional activities, and available learning content including structured 
exercises, activities, and assessments. To successfully learn the required content contained within this 
module, numerous learning objectives from the different CODIAC modules were analyzed. Supporting 
knowledge, skills, and abilities were defined, cataloged, and decomposed into their own competency 
network. Table 7 on the following page shows the how the ELOs and TLOs were structured for the original 
course. These were augmented with ELOs and TLOs from other related courses. The modified CODIAC 
course used for the 2018 demonstration was supported by other educational resources from across the 
services including the following: 


e CODIAC POL including several hours of edited video — permission received from JSJ7 
e Combat Hunter eLearning course — permission received from USMC 


e PercepTS Small Unit Leader Kit — Permission received from U.S. Navy 
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Table 7. CODIAC ELOs and TLOs. These ELOs and TLOs derived from the existing course structure presented a 
guide for identifying, reviewing, and aligning content with an overall CODIAC Competency Framework. 


Module Name Description 
TLO1 Left of Bang This module is about positioning to out-fight the enemy, specifically out-think them too. At the 
end of this module, learners will explain the importance of operational tempo, of making sense 
of the situation faster than our opponents to anticipate their actions and be able to act “left-of- 
bang.” 

ELO1.1 Define Consequences Define consequences of being reactive versus proactive on the battlefield 

ELO1.2 Incident Timeline Explain incident timeline and Left-of-Bang. 

ELO1.3 Benefits Describe benefits. 

ELO1.4 Response Describe when and how to respond Left-of-Bang 

TLO2 The OODA Loop At the conclusion of this unit, trainees will be able to explain the cognitive processes associated 
with the OODA-Loop, including how these processes are impaired by stress and how decision- 
making limitations can be mitigated, such as employing the five combat multipliers. 

ELO2.1 Cognitive Processes Explain cognitive processes associated with the OODA loop. 

ELO2.2 Define Steps Define steps in the OODA loop. 

ELO2.3 Decision-making Given various scenarios, explain the process for OODA loop decision-making. 

ELO2.4 Behavioral Cues Effect of behavioral cues on OODA loop and decision-making 

TLO3 Baseline + Anomaly = This module introduces the concept of baseline and anomaly, and how together they require a 
Decision decision. Upon completion of this module, learners will be able to describe baseline, an anomaly, 
and how anomalies to the baseline require a decision. 

ELO3.1 Describe Baseline Describe the concept of a baseline. 

ELO3.2 Describe Anomaly Describe the concept of an anomaly. 

ELO3.3 Explain Decision-making Explain why a decision is required when there is an anomaly 

ELO3.4 Decision-making Process Given various scenarios which include anomalies, explain the process for decision-making. 

TLO4 Reading the Human Terrain This unit introduces techniques for interpreting human behavior cues. At the conclusion of this 
module, learners will be able to describe the six domains of profiling in detail and will be able to 
demonstrate the six domains’ use. 

ELO4.1 Biometrics Describe biometrics; list several biometric cues and their interpretations. There are 11 specific 
biometric cues discussed. We may want to decompose further. 

ELO4.1.1 Explain Biometrics Explain the biometrics domain. 
ELO4.1.2 Interpret Biometric Cues Given specific biometric cues, explain what they might mean. 
ELO4.1.3 Apply Biometrics Describe how to apply biometrics in the battlespace. 
ELO4.2 Kinesics Describe Kinesics; list several kinesic cues and their interpretations. There are 8 specific kinesic 
cues discussed; we may want to decompose further. 
ELO4.2.1 Explain Kinesics Explain the kinesics domain. 
ELO4.2.2 Interpret Kinesic Cues Given specific kinesic cues, explain what they might mean. 
ELO4.2.3 Apply Kinesics Describe how to apply kinesics in the battlespace. 
ELO4.3 Proxemics Describe proxemics; list several proxemic cues and their interpretations. There are 11 specific 
proxemic cues (4 groups) discussed. We may want to decompose further. 
ELO4.3.1 Explain Proxemics Explain the proxemics domain. 
ELO4.3.2 Interpret Proxemic Cues Given specific proxemic cues, explain what they might mean. 
ELO4.3.3 Apply proxemics Describe how to apply proxemics in the battlespace. 
ELO4.4 Geographics Describe geographics; list several geographics cues and their interpretations. There are 5 specific 
geographics cues discussed. We may want to decompose further. 
ELO4.4.1 Explain Geographics Explain the geographics domain. 
ELO4.4.2 Interpret Geographics Cues Given specific geographic cues, explain what they might mean. 
ELO4.4.3 Apply Geographics Describe how to apply geographics in the battlespace. 
ELO4.5 Atmospherics Describe atmospherics; list several atmospheric cues and their interpretations. There are 5 
specific atmospheric cues (4 groups) discussed. We may want to decompose further. 
ELO4.5.1 Explain Atmospherics Explain the atmospherics domain. 
ELO4.5.2 Interpret Atmospherics Given specific atmospherics cues, explain what they might mean. 
ELO4.5.3 Apply Atmospherics Describe how to apply atmospherics in the battlespace. 
ELO4.6 Heuristics Discuss heuristics and explain how heuristic matches are made. 
ELO4.6.1 Explain Heuristics Explain the heuristics domain. 
ELO4.6.2 Explain Dangers of Heuristics | Explain the dangers of the heuristics domain. 

ELO4.7 Icons, Colors, and Symbols List and interpret the relevant icons, colors and symbols in one’s own area of operation. There 
are 4 specific image-types discussed; with colors having 7 different significances. We may want 
to decompose further. 

ELO4.7.1 Explain Iconography and Explain the importance of iconography and symbolism. 
Symbolism 
ELO4.7.2 Interpret Icons and Symbols Give examples of specific icons, color and symbols, and their meanings. 
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Figure 15. CODIAC Competency Framework — The CODIAC course structure provided the foundational ELOs 
and TLOs that informed the creation of a related Competency Framework. This is an example Competency Framework 
created around the Combat Tracking TLO. 


4.2.2 CODIAC Competency Framework 


As shown Figure 15, the competency framework developed to support the 2018 TLA Test and 
Demonstration was largely defined by each competency’s many-to-many relationship with other 
competency objects. This is complicated by the need to weight the inferred relationship between 
competencies which are different depending on the context (e.g., job, role, and periodicity) of how they are 
applied. The discussion highlighted some of the inherent challenges in migrating to a competency-based 
education system. 


The CODIAC ELOs and TLOs were written succinctly so they can be measured and assessed; however, it 
was determined that the simple mapping of ELOs and TLOs to competencies did not adequately reflect the 
entirety or potential of what a competency model should include in its definition. Competencies and 
learning objectives are two related educational terms that can create confusion. Competencies define the 
applied skills and knowledge that enable people to successfully perform their work while learning 
objectives are specific to a course of instruction. Learning objectives describe what the learner should be 
able to achieve at the end of a learning period. There may be more than one measurable learning objective 
defined for a given competency. While competencies and learning objectives can be written similarly, a 
competency is obtained or developed during the process of learning and is intended to describe the desired 
knowledge, skills, and behaviors we want students to walk away with after taking the course. 


Due to time constraints, three cross-cutting competency frameworks comprised of 234 interrelated 
competency objects were created to illustrate how different competencies can be inferred from different 
assessment activities. Three clusters of competencies were created to include Combat Awareness, Combat 
Profiling, and Combat Tracking. As shown in Figure 16, CaSS was used to encode the frameworks. As 
assertions are made from existing assessments, they are applied to the relevant competencies included in 
these three frameworks via a unique identifier for each competency. For example, as a student evaluates an 
image, the specific act of analyzing the image might demonstrate proficiency with the proxemics ELO that 
traces back to both Combat Awareness and Combat Profiling. 
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Figure 16. Competency Framework — The CaSS Competency Framework Editor was used to encode the CODIAC 
competency frameworks used within the 2018 TLA Test and Demonstration. CaSS was used to define the basic 
elements of each competency and its relationships to other competencies. Each competency object has a unique 
numerical identifier that was referenced using the LRMI alignment object. 


4.2.3 CODIAC Content Discovery and Alignment 


One goal of the 2018 TLA Test and Demonstration was to go beyond traditional DL content to demonstrate 
the breadth of learning activities that can be connected into the TLA. As shown in Table 8, the 2018 TLA 
Learning Record Providers span the range of learning activities from traditional eLearning content (e.g., 
SCORM courses) to emerging technologies like serious games and mobile learning applications. 


After encoding the Competency Frameworks, the content obtained from CODIAC, Combat Hunter, and 
PercepTS was reviewed, cataloged and aligned to each competency included in the framework. An initial 
assessment revealed existing content that included video-based lectures, reading assignments, instructor 
activities/handbooks, assessments/quizzes, interactive web-based training lessons, and written, scenario- 
based exercises. A gap analysis was performed to determine how many learning activities were available 
to support each competency object. It was discovered that most of the available DL content and assessment 
activities were appropriate to support knowledge and comprehension, but were not adequate for 
demonstrating the application, analysis, or synthesis of the required knowledge, skills and abilities 
associated with many of the competencies. These were necessary to support the novice, intermediate, and 
advanced levels of mastery within each competency. 


Further, to differentiate between recommendations, numerous learning activities and assessment activities 
were required for each identified competency in the framework. A search was performed to identify 
additional formal and informal learning activities including non-digital content (e.g., Combat Hunter card 
game), serious gaming applications, and scenario-based live exercises. Other courses were also identified 
from the Army Distance Learning Program (TADLP) and Joint Knowledge Online (JKO). These courses 
ultimately were not used; however, the ADL Initiative staff did create six serious game vignettes using the 
Unity3D game engine. Numerous reference materials including articles, handbooks, lessons learned, and 
other materials were also collected and mapped to each competency. 
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Table 8. 2018 TLA Learning Record Providers — The 2018 learning activities included many different types of 
learning content that were instrumented with xAPI statements that are then transmitted to an LRS. 


TLA Learning Short Description Related 
Record Provider Standards 


(LRP) 


Live Exercises Instructor-led exercises and group activities were used as Capstone XxAPI 
assessments for receiving a badge and to assess for other higher- 
level competencies. 

Moodle™ LMS MarineNet Combat Hunter e-Learning Course: Three SCORM  SCORM, xAPI 
lessons that consisted of a single large Flash® file that encapsulated 
all interactions and behaviors. These files had very limited reporting 


capabilities. 
Moodle™ LMS Multiple choice and short answer quizzes implemented throughout xAPI 
the program of instruction. 
PALMs Perceptual Adaptive Learning Modules. xAPI 
PEBL eBook Content in the form of an e-publication has the characteristics of a | xAPI, 


book but is augmented by the technologies included on each device. _ePub3/ADB 
Sero! Concept Maps = Open Source Concept Map toolkit used to create capstone exercises | xAPI, 
to demonstrate understanding about key concepts. Used for 


assessment. 
Static Content An ADL Initiative-developed viewer that allows a learner to self- | xAPI, 
Viewer report whether they’ve completed viewing/reading assigned 


content. A big limitation of this platform is that it is only able to 

collect initializes and completes data. 
Unity3D Serious Situational Judgement Trainers: Content delivered in the context of | xAPI, Serious 
Game a digital, interactive judgement trainer where scenarios are Games xAPI 

presented to each learner and they are assessed on their responses. | Profile, WebGL 
xAPI Video Player Web-based utility that plays any number of video formats and tracks xAPI 

viewer usage (e.g., pause, play, skip, focus, etc.) 


To measure the transfer of learning, a pre-test and post-test were created. Each test consisted of multiple- 
choice questions, matching exercise, true/false questions, and free-response short answer questions. Other 
assessment activities were integrated into the various activity providers, and in some cases, a single 
assessment was aligned with multiple competencies. A minimum of six assessment activities were desired 
for each competency with at least two of those assessment activities capable of evaluating the higher orders 
of Bloom’s taxonomy. Situational Judgement Trainers and concept maps were created to allow learners to 
demonstrate their understanding of different competencies. Live, scenario-driven exercises were performed 
with an instructor as a capstone event to receive a badge for each of the three competency frameworks 
(Combat Awareness, Combat Profiling, Combat Tracking). 


4.2.4 Instrumenting CODIAC Content 


CODIAC learning activities were initially aligned to 
competencies using an Excel spreadsheet. Decisions had to be | A working group was 
made about which TLA Activity Providers will deliver which originally established to 
content, where different pieces of content will reside, and how focus on this topic; 
each piece of content will be launched, tracked, and managed over 


however, this working 
the course of the one-week test and demonstration. 


group was absorbed by the 
A primary focus of the 2018 TLA Test and Demonstration was to Content Demonstration 
demonstrate the instrumentation, collection, aggregation, ; 
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analysis, and visualization of learning data. To meet this goal, a data strategy was created to define how 
each activity would be described using metadata and an xAPI profile was created to enable a shared 
vocabulary for consistently tracking learner performance across all activities. Metadata is used by FLUENT 
to help inform recommendations to the learner and the CaSS to help inform the meaning behind different 
assertions being processed. Mandatory and optional xAPI statements were created for each media type used 
in the 2018 demonstration. These were derived from the types of performance data that each media type 
can generate, the roles of each user that needs to visualize this data, and the type of data that needs to be 
generated to support each user. 
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Figure 17. xAPI Statement Mappings — The xAPI working group began identifying basic xAPI statements that will 
be required across all media types. Additional statements will be needed as the different types of content/media are 
confirmed for use in the 2018 demonstration. 

As shown in Figure 17, numerous strategies were discussed for how and when to instrument the different 
learning activities. Dashboards and data analytics strategies ranged from instrumenting learning activities 
with xAPI statements to the fusion of data from other TLA sources. Other data types include competency 
data and Learner Profile information created by CaSS, metadata for content/activities, and learner inference 
data derived from FLUENT. A TLA data requirements document was authored to clearly define all activity 
tracking requirements for each learning activity. This document provided guidance for those implementing 
xAPI to ensure data interoperability across learning activities, but also provided a foundation to inform the 
types of dashboards that would be available in the 2018 event. 


An overarching xAPI profile was created for the 2018 TLA Test and Demonstration that was comprised of 
numerous other xAPI profiles and a shared vocabulary that allowed disparate TLA services and content 
providers to track data consistently, facilitating a common language for analytics and visualization services. 
However, it was not implemented consistently across all 2018 activities and components. The xAPI actors 
were defined by the participants UUID (assigned by Keycloak™). Statement templates were created for 
specific learning activities and experiences that were delivered to learners. Client-based services like the 
video player or PeBL integrated quickly; server-based services like PALMS, Unity3D, and Moodle™ 
required special helper services to facilitate full integration. 
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Figure 18. TLA Metadata Spreadsheet — An Excel spreadsheet captured the 23 fields used to describe each learning activity. The Activity Name and description 
were used verbatim by the recommender’s interface to provide the learner information about how the activity applies to the competencies it is aligned with. As 
learning activities were added to the spreadsheet, a script pulled data from the relevant cells and automated the process of creating an LRMI formatted JSON file. 
The JSON file was uploaded to the Activity Index which acted as a common repository that all TLA components used to understand relevant information about 
learning content and activities. 
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Using the LRMI format, the 2018 metadata schema included 23 properties describing the activity, including 
its location, associated competency alignments, associated thumbnail, and a 140-character description of 
the learning activity. The metadata files also included human-readable text fields (e.g., name and 
description) that FLUENT used to display to the learners as part of the recommender interface. As shown 
in Figure 18, the approach was a labor-intensive process that required human data entry into an Excel 
spreadsheet that included all metadata fields. 


This data was then exported and brought into a script that could generate JSON files that were conformant 
with the LRMI specification. The JSON files were aggregated into an Activity Index which was resident 
inside the FLUENT boundary. The Activity Index was used by FLUENT content filtering algorithms to 
make basic recommendations. CASS also used the Activity Index each time it received an assertion from 
an activity. CaSS would use the activities metadata to align to a specific competency and to inform mastery 
estimates. 


4.2.5 Sequencing and Delivery 


While the original CODIAC course was an established program of instruction, the 2018 Test and 
Demonstration took other courses such as Combat Hunter and PercepTS and merged this content with other 
available content that was never considered in the original development of the CODIAC course. This 
presents the challenge of reassembling the content in a way that makes pedagogical sense to learners and 
instructors. Often, this is because the fundamental elements of instruction (objectives, content, instructional 
events, activities, and assessments) are not aligned with sound learning principles or instructional strategies. 


While the CODIAC course was being migrated into the TLA competency framework, instructional 
designers reviewed the domain and found that the course contains both well-defined and ill-defined tasks. 
Table 9 and Table 10 show a breakdown for how different instructional activities will be divided for the 
2018 TLA Test and Demonstration. 


Table 9. Well Structured Tasks. [Instructional framework and sequencing of activities for well-structured tasks and 
problems. 


e = Tell me (verbal information, concepts, procedures, principles) 

e Show me (demonstration) 

e Copy me (demonstration with active participation) 

e Locate, name, identify, list, match (information, concepts, procedures, 


1. Presentation 


Poe ntactee principles)followed by correct answer or corrective feedback 
e Discuss verbal information, concepts, procedures, principles 
e Do next or you do it with correct answer feedback 
e Given conditions, predict consequences with correct answer feedback 
e Given outcome, specify conditions with correct answer feedback 
3. Support e Motivational, cognitive, and metacognitive adaptions 


e §=Corrective feedback 
e Explanation of what happened and why 
e _ Discussion of verbal information, concepts, procedures, principles 
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Table 10. Ill-Defined Tasks. Problem-centered framework and sequencing of activities for ill-defined tasks and 
problems. 


1. Authentic e =I Il-structured learning tasks (e.g. CODIAC Unit 5, Lessons 5001-5006 — Meaning of cues, 
Broblens image, 5008 — Profile environments) 

e Practical experiences (e.g. CODIAC Unit 8, Lessons 8005, 8006, 8007, 8010, 8013) 

e = Practical Applications (e.g. CODIAC Unit 9, Lessons 9001-9010) 

e Show me (demonstration) 

e Copy me (demonstration with active participation 

e Do next step or you do it with correct answer feedback 


2.Demonstration 


3.Practice ; a ; . 
e Given conditions, predict consequences with correct answer feedback 
e Given outcome, specify condition with correct answer feedback 
ASupser: e =Tell me and Show me (verbal information, concepts, procedures, principles covered in 


unit 1-7 and lessons 8001-8004, 8008, 8009, 8011, 8012) 

e Locate, name, identify, list, match (information, concepts, procedures, principles) 
followed by correct answer or corrective feedback 

e Discuss (or videotaped discussion of) verbal information, concepts, procedures, and 
principles covered in Unit 1-7, and lessons 8001-8004, 8008, 8009, 8011, 8012 

e Motivational, cognitive, and metacognitive adaptions 

e Corrective feedback and explanation of what happened and why 


The FLUENT recommendation engine was developed to accommodate multiple instructional frameworks 
to guide recommendations according to established instructional strategies that are commonly used today. 
This allows future implementations to tailor the instruction according to the domain being taught and other 
considerations where different instructional strategies are desired. For the 2018 TLA Test and 
Demonstration, FLUENT was instrumented with two frameworks that were best suited to deliver 
recommendations for which content a user should encounter next. Figure 19 and Figure 20 below show 
how content was delivered to support well-structured and ill-defined tasks. 


Task Show KSAs 
Notification for Job Task 


Pretest the 
Job Task 


Deliver 
Activity 


Check on Main Loop 


Learning 


Collect Paradata/ 
Activity Feedbac 


Gate to 
Progression 


1. Same Category 
2. Next Category 
3. KSA Done, Next KSA 


Infer Task 
Proficiency, 
KSA Status 


Select Target 
KSA 


Select Activity 
Category 


1. Recall Knowledge 
2. Interactive Demo 
3. Skill Practice 


Post-test the 
Job Task at 80% 


Figure 19. Well Structured Tasks — This represents an instructional framework for delivering content in support of 
well-structured tasks. Content is presented in a linear fashion where knowledge, Skills, and Abilities (KSAs) are 
structured hierarchically and the user cycles through learning content while using the checks on learning to progress 
to each consecutive level. 
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Figure 20. Ill-Defined Tasks — This represents an instructional framework for delivering content in support of ill- 
defined tasks such as understanding the meaning of cues inside different images. For these types of tasks, the user 
cycles through content and a learner behavior model is updated and used to determine the appropriate content the 
user should encounter next. 


4.3 TECHNICAL INTEGRATION 


After an initial registration with Keycloak™, the FLUENT recommendation engine provided the primary 
interface to the learner through a browser based HTML5 application. Upon log-in, users were sent to the 
home page seen below in Figure 21. This is the primary interface for the students and offered a few different 
methods for accessing content. Basic functionality included the ability for each learner to set his or her 
goals and visualize progress through an interactive skill tree. Based on selected goals, FLUENT would 
direct learners to different LRPs including the Moodle™ LMS, PeBL, PALMs, Sero!, and live exercises. 
Each LRP was capable of delivering different learning activities that were tracked and used to make future 
recommendations. 
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Figure 21. Home Page of 2018 TLA Test and Demonstration FLUENT Interface. 
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Instructional content had to be presented in a logical and intuitive way that maximized our ability collect, 
analyze and display learning data to the relevant target audience. By instrumenting an array of disparate 
content, the 2018 TLA Test and Demonstration generated a large amount of inter-connected human 
performance data (individuals and teams) that could be used to demonstrate the potential of the TLA for 
observing learner progress across learning modalities. The recommended activities generated by the 
FLUENT system were based on the goal selected by the user on the screen shown in Figure 22. For this 
event, instructional content was organized into skill trees with three top-level nodes (badges): Combat 
Awareness, Combat Profiling, and Combat Tracking. This structure was solely for content organization, 
not to restrict users to a predefined path. Learners could begin at any level of a skill tree and there were no 
restrictions on what goals learners could select or how often they could change goals. 
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Figure 22. FLUENT Interface Showing the CODIAC Skill Tree Structure and Goal Selection 


As students worked through the content, xAPI statements were generated to record their interactions with 
the system. These statements were captured in the Learning Locker LRS. One LRS was set up to capture 
learner actions (e.g., answering a question), while another was set up to capture system actions (e.g., making 
a competency assertion). Each TLA component published their system log files via xAPI statements to 
store a ledger of what was going on internal to each system. xAPI system logs were housed in a separate 
LRS and were used to provide additional insight into the computational resources and data dependencies 
in the 2018 TLA Reference Implementation. In addition, non-xAPI learner inference data were stored in 
the CaSS. This resulted in three data sets: 


e Learner data: a 77 MB JSON file with 57,381 xAPI statements 
e System log data: a 1.8 GB JSON file with 353,746 xAPI statements 
e Learner inference data: a 90 MB JSON file (non-xAPI format) 


xAPI Data Structure: The xAPI is a technical data specification that enables flexible and customizable 
tracking of learner performance and behaviors across different learning activities. The xAPI statements are 
constructed in a JavaScript Object Notation (JSON) format with three important properties: Actor, Verb, 
and Object. In sequence, these pieces of data produce a clear recording of the learning experience. These 
data can also be cross-referenced with performance data to map the training to performance. The xAPI 
specification is open-source and maintained by the ADL Initiative. In addition to the three data fields 
mentioned above, the following properties were also used in the demonstration: 
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e ID: a unique identifier for each xAPI statement 

e Version: the statement’s associated xAPI version 

e Timestamp: when the event described by the statement occurred 

e Stored: when the statement was recorded by the LRS 

e Result: further details representing a measured outcome (e.g., assessment score) 
e Authority: agent or group behind assertion; verified by LRS via authentication 
e Context: further details of recorded event (e.g., altitude of flight simulator) 


Competencies and Badging: Badges represent a way of acknowledging achievements or skill acquisition 
in a granular fashion. The CODIAC course was organized into skill trees with three top-level nodes: 


e Combat Awareness 
e Combat Profiling 
e Combat Tracking 


These nodes were deemed badges and arbitrarily given internal codes 1, 2, and 3, respectively. Expanding 
each node revealed more layers of content, up to a maximum of five layers (e.g., competency code 
3.1.3.B.3). Each competency’s unique ID was a URL that contained further text information about it, 
including its weight, description, and internal code. 


Learner Inferences: CaSS used each learner’s data to estimate their proficiency for each node of the skill 
tree. This information was stored in JSON format, but it did not follow the xAPI structure. As such, these 
data were not logged by an LRS but were instead stored locally in CaSS. These data included: 


e Number of attempts and timestamp of last attempt for each activity 

e Mastery estimates (novice, intermediate, expert) for each competency 

e Mastery probabilities, assertion source, and timestamp for each competency (the CaSS assertion 
processor maintains a list of systems that can provide assertions) 


Activity Index: The Activity Index was stored as an LRMI-formatted JSON file located inside the 
FLUENT application boundary. The metadata drives the filtering algorithms used in the prioritization of 
recommended learning activities. The metadata model also included paradata, assertions, analytical data, 
identities, and reputations that flow through the distribution network. CaSS also has a service that goes into 
the Activity Index and retrieves a mapping from the xAPI assertion statement to the appropriate metadata 


for the activity generating the statement. Each competency has a unique identifier in the competency 
framework. This is referenced in the metadata through the LRMI Alignment Object. 


Figure 23. SWEG(A) Classroom at Fort Bragg — The classrooms for the TLA 2018 Test and Demonstration were 
reconfigurable and allowed the team to support all participants at once for completing the initial pre-test and surveys. 
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4.4 TLA TEST EVENT AT FORT BRAGG 


Learners valued the content, the team collected an enormous quantity of data, event administration ran 
smoothly and timely, no system outages occurred, and the vision for the future learning ecosystem was 
realized to an enthusiastic audience. The event itself took place from August 13-17, 2018 at Fort Bragg in 
Fayetteville, North Carolina. As shown in Figure 23, the 2018 TLA Test and Demonstration was delivered 
in collaboration with the John F. Kennedy Special Warfare Center and School (JFKSWCS), with 
participants from its Army SWEG(A). Roughly 60 volunteer active duty personnel from SWEG(A) 
interacted with the TLA Reference Implementation for 14 hours over five days. 


On the first day, participants were briefed on the purpose of the event and provided an overview of the tools 
and technologies they would encounter throughout the week. Upon completion of the briefings, participants 
signed the inform consent forms and began completing a survey that collected demographic information 
about each participant. Upon completion of the surveys, each participant took a pre-test that was intended 
to measure their current state of proficiency across the 234 competencies identified for use in this course. 
The pre-test was graded, and the results were used to initialize the participant’s Learner Profile so that the 
different TLA systems could tailor recommendations based on each individual’s current level of 
understanding. The pre-test results were also compared to the results of a post-test taken after the event to 
measure the immediate transfer of learning after taking the course. 


AS participants navigated through the course, observers continually collected data about what each 
participant was working on, and several focus groups were held after the course to poll participants on 
various aspects of their experience in the course. Using this data, coupled with the digital data collected 
throughout the event, seven areas of the TLA were evaluated: 


1. Functionality: TLA features and proposed benefits were documented, and the reference 
implementation was assessed against the defined functional requirements. As participants 
interacted with the system, basic system metrics, user behaviors, and system behavior were 
gathered to determine functional performance and whether design goals were met. Some of the 
measures used for assessing functionality included a comparison of systems actions taken as a result 
of corresponding event data and the comparison of recommendations to competency data per user. 


2. General System -ilities: Related to the functionality assessment, the system’s general technical 
performance was captured; this includes criteria such as latency, composability, technical reliability, 
and modularity. System interruptions, downtime, and stress load failures were also monitored. 


3. Specific System -ilities: This involves documentation and assessment of the idiosyncratic technical 
performance criteria, such as how varied learners’ trajectories are from one another (i.e., system 
adaptability). System logs about operational performance were also reviewed to help understand 
how and why different actions occurred. 


4. Usability (End Users): This assessment included learners’ satisfaction, engagement, and 
subjective experiences using the system. Data was captured using existing instruments (1.e., System 
Usability Scale [Brooke, 1996] and User Experience Questionnaire [Turner, 2011]). 


5. Usability (Developers): This assessment focused on the satisfaction and experience of those who 
interact with the system for design and development purposes, such as content developers and 
instructional designers; this involved both the reference implementation and the viability and 
quality of its technical documentation in terms of clarity, functionality and diffusion. 
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6. Learning Outcomes: Although the transfer of learning was not a focus for this test, the system’s 
learning potential was assessed to provide a baseline for future experiments. This was determined 
primarily by measuring learning gains using a pre/post-test design. 


7. Cost Factors: Finally, initial development cost data (to eventually inform return on investment 
analyses) was captured from the system designers and developers. Data collection includes both 
qualitative and quantitative data. 


Upon completion of the course, all collected date, including the results from the surveys, pre and post-tests, 
and the three primary TLA data stores were cleansed and packaged in support of a TLA data hack-a-thon. 
This event invited engineers, data scientists, and others to get an in-depth understanding of what data was 
collected and made available during the 2018 TLA Test and Demonstration event. The hack-a-thon 
provided detailed results about the potential for how learning data may be used to power data-driven 
dashboards that support a variety of different roles and purposes across the continuum of learning. 


4.4.1 Logistics and Administration 


Leading up to the August 2018 event, the ADL Initiative worked with SWEG(A) and IDA to create surveys, 
orientation briefings, and coordinate the exchange of materials required to obtain all the various permissions 
for holding the event. The ADL Initiative coordinated staffing, travel, hardware/software implementation, 
and overall management of the event. IDA created the research plan, and SWEG(A) provided 30 desktop 
computers, 3 classrooms, student participants, and instructor/observer participants as needed to help 
facilitate the 2018 demonstration. Prior to the start of the demonstration, the ADL Initiative staff and 
vendors spent the week with SWEG(A) staff preparing the classroom, computers, and devices for the 
demonstration. The team partitioned the classroom into two areas: a large area for learners and their 
working areas, and a smaller area for developers and support staff. A small meeting room was also provided 
to support focus groups and other ad hoc meetings. 


The Research Plan submitted through IDA followed all policies required for the protection of human 
research subjects. With concurring approval by the Department of Defense Personnel and Readiness 
Research Regulatory Oversight Office, this project received an exemption. Participants were divided into 
groups of 30. This allowed for a 4-hour class in the morning and one in the afternoon. SWEG(A) confirmed 
participant availability and provided all the primary points of contact for JEKSWCS and Fort Bragg policies, 
permissions and access. Participants worked in a computer lab with Internet connected desktop computers, 
iPad devices, and WIFI access for iPad Internet connectivity. Public Internet was accessed through Fort 
Bragg’s LangNet. SWEG(A) reviewed all instructional content and ran scans on all network-connected 
hardware to monitor ingoing/outgoing network traffic. 


4.4.2 Hardware and Network Connectivity 


The TLA’s AWS infrastructure met all requirements. With over 30 concurrent users streaming video, 
reading PDFs, and using Moodle™, the team and learners observed no system outages. Networking, 
memory, and CPU metrics were well below their limits, rarely reaching even 50% capacity. Rather than 
configuring 30 laptops individually, the SWEG(A) engineers requested the team provide a single, pre- 
configured master hard drive that could be cloned across all classroom computers. SWEG(A) used their 
Master Drive Cloning System to rapidly deploy preconfigured operating systems, applications, and settings 
on each desktop computer. The lead time for this system is 30-60 days (ideal). SWEG(A) delivered a single 
desktop computer at the end of June from the 25" to the 28". The ADL Initiative engineers received the 
desktop and configured it to run the required software components. 
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With 30 concurrent learners using two devices each, network latency and bandwidth limits were a concern. 
The team used two Netgear Nighthawk routers broadcasting two frequencies** each, allowing up to 128 
simultaneous connections — satisfying connectivity requirements for the learner devices and allowing 
network access for developers and administrators. After setting up the routers, network connections for 
each device were configured with fallbacks to the developer network if necessary. Devices typically 
maintained their connection, but wireless settings required the team confirm connectivity each morning. 
For security reasons, SWEG(A) did not broadcast the wireless networks, so the team established each 
connection manually across devices. Any loss of connectivity was only temporary as the backup network 
took effect, but a building-wide power outage did impair connectivity for roughly 30 minutes during one 
session. 


Computers, iPads, and peripheral hardware functioned without issue. The desktops were preconfigured with 
the software preloaded through SWEG(A)’s cloning process. Minor changes were made to the software as 
some TLA components were updated after the cloning process. On the desktops, very few issues arose. 
System permissions from SWEG(A) prevented the team from resolving some issues with browser caching”, 
so the team changed the learners’ web browsers from Google Chrome to Mozilla’s Firefox during the 
demonstration resolve this. Initial system usage on the first day (Tuesday) uncovered a performance issue 
with the LRS responsible for collecting system logs. The server was upgraded and its corresponding DNS 
registration by midnight. The machine resumed function Wednesday morning. 


4.4.3 Pre-Tests, Post-Tests, Surveys, and Focus Groups 


Upon arrival Tuesday morning, all students gathered to sit 
through an orientation brief, signed their informed consent 


Orion tation T oe. 


14-17 Aug 2018 


forms, and completed a series of surveys. Upon entering 


the classroom, SWEG(A) staff randomly assigned a UUID WI 23 FID TD FD 


to each participant to ensure future anonymity in analysis 


Project KO DBR Test #1 DBR Test #2 Marketplace Hardening 


and reporting. Log-in accounts had previously been created | 


with the UUIDs and only SWEG(A) and IDA had access 
to Soldier identities. As shown in Figure 24, the 
Orientation provided an overview of the multi-year 
initiative and walked participants through the intended 
purpose and justification for this research. The CODIAC Figure 24. Orientation Briefing — The 20/8 


course was introduced and the various delivery platforms | TLA Test and Demonstration is the culmination 
of the second spiral of Design Based Research. 


were described to show participants how to use the system. 


The first survey was a Basic Demographic Questionnaire consisted of 16 items including name, gender, 
age, service, duty station, office, assignment, military grade, military occupational specialty code, education, 
and background. The target populations for this study are U.S. Army active duty or reserve military 
personnel with military grades spanning mid and senior enlisted, non-commissioned officers, warrant 
officers, and non-field grade officers. They are stationed at Ft. Bragg and are all under the US Army Special 
Operations Command (USASOC) with Psychological Operations and Special Forces military occupations 


48 Each router broadcast its 2.4 GHz and 5 GHz frequencies. 

* Caching is a process employed by modern web browsers to reduce page load times. These browsers keep local 
caches of certain files from webpages that they expect to rarely change (like style sheets and images). While this 
improves user experience, clearing these caches often requires administrator permission on the machine itself. 
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represented. The age span is approximately 20-26 years with gender skewed mostly toward males. A User 
Experience (UX) Questionnaire was also presented for collecting data about system usability and the user’s 
experience interacting with the system. A 24-item UX Questionnaire was developed to collect data on 
usability and user experience. These surveys helped evaluators understand more about the participants and 
for stratification in analysis. 


Researchers were also on site to collect observational data of the interactions in correlation with the 
collection of system performance and behavioral data from TLA systems over the interaction period. These 
data should enable a unit of analysis at the individual level as well as the group and provide descriptive data 
on learner individualization and system level performance. 


To capture initial and final knowledge of the subject matter, the team administered pre-tests and post-tests. 
Pretest scores were used to seed the initial user competency values in the system. Though originally 
planning to use Moodle™ for serving these assessments, the team utilized paper copies because the final 
pre- and post-test included short answers and essay questions which required manual grading outside of 
Moodle™. Two separate tests were developed; one for the pre-test and one for the post-test. All learners 
went through and took their pre-tests at the same time Tuesday morning. Eight team members with 
knowledge of the CODIAC material graded each test manually. As the experts finished grading these pre- 
tests, results were transferred to an Excel spreadsheet, and run through a script to import those results into 
CaSS to initialize the participant’s TLA Learner Profile. Each paper test was digitally scanned upon 
completion and again after grading was completed. The lack of automation required considerable resources. 


To understand more about each participant’s context of use, insight into thoughts and feelings about their 
experience, and other individual/group dynamics, post intervention focus groups were held. The focus 
group protocols called for a short social and warm-up period, followed by the open-ended questions and 
conversation, and finally a wrap up. Questions were created ahead of time and were intended to draw out 
experiences and perceptions concerning expectations of learning, expectations of interaction, actual 
experiences of interaction, contrasts between expectations and actual system performance, and participants’ 
feelings about use. Focus groups were closed between the researchers and the participants and will be 
recorded for analysis. 


Lastly, the research team recorded learner, administrator, and engineer feedback about the event through 
surveys and focus groups. Small groups of learners were led through focus groups while other the ADL 
Initiative staff and IDA discussed the event with SWEG(A) administrators and instructors. Interviews were 
also held with the ADL Initiative and vendor engineers to document their experiences. In addition to face- 
to-face focus groups, participants also completed paper surveys for both administrative and engineering 
efforts. 


4.4.4 Learner Management and User Experience 


Prior to the event, the ADL Initiative created 90 TLA accounts to be used by learners at Fort Bragg. Each 
participant received login credentials for a UUID they kept and used throughout the event, allowing the 
system to track their individual performance without collecting any Personally Identifiable Information 
(PII). Learners were asked to preserve their login credentials between days, although these credentials could 
be given to a learner upon request. 


After login credentials were distributed and all in-processing activities were complete, instructions were 
provided on how to access and use the system, learners logged into their designated accounts and began 
using the FLUENT User Interface (UI). Learners will interact with the CODIAC content for approximately 
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four hours per day over a four-day period with the first and last day limited to two hours to allow for in- 
processing and out-processing the participants. Fixed groups of 30 participants alternated at designated 
times throughout the week. When using the FLUENT UI, learners had difficulty knowing which 
competencies had not been achieved due to a scaling issue*’. By default, the tree view expanded nodes with 
a fixed font size and did not reconcile tightly grouped elements for visibility, making large groups of 
activities unreadable. 


As the TLA is designed to empower personalized learning, everyone experienced a different learning path 
through the CODIAC content. Overall, the system and content kept learners engaged and allowed them to 
progress through the content in a logical fashion. Since the learner was the sole pivot point for all 
recommendations, they alone chose their learning goals and corresponding activities. While many learners 
began with the SCORM content (presumably because its relationship to so many competencies almost 
guaranteed its recommendation), their activity paths began diverging after those were completed. 


Participants viewed the relevance of CODIAC content as a positive experience and especially called out 
the live activities and capstone exercises as having value to their near-term learning goals. It is important 
to note that the human readable descriptions included in the metadata for each activity and the custom 
thumbnail images also contributed to a positive learner experience by explaining why each activity was 
related to different competencies. The research team observed some learners intentionally change their 
goals to seek out an interesting or fun activity being performed by their neighbor. 


4.4.5 Data Collection and Management 


Automated data collection efforts produced a significant quantity of data. The system collected over 
450,000 xAPI statements during the 4-day event, which were anonymized and distributed for the iFEST 
hack-a-thon. Approximately 400,000 of these were generated through the system log LRS and the other 
50,000 were generated through learning activities to reflect participant behaviors. The “2018 TLA Data 
Requirements” document defined the meaning, structure, and requirements for the xAPI statements 
produced within the TLA. Other than xAPI statements, the system captured each learner’s final competency 
profile, their activity history, and statistically significant pre-test and post-test differentials. Additionally, 
the surveys and focus groups provided a wealth of objective information about TLA usage, user interface, 
instructional content, value proposition, and overall user experiences. All data was collected and put under 
positive control by the ADL Initiative and IDA at the end of the event. 


An objective third party evaluation of the raw data by an ADL Initiative vendor that was not part of the 
2018 event concluded that the dataset collected at Fort Bragg had limited value for the purpose of true 
analytics. This is a result of many factors including inconsistent usage of terms due to inconsistent usage 
of the TLA’s xAPI profile. Some of the content providers used different verbs or did not include the result 
extensions as outlined in the Data Requirements document. The result of numerous misalignments between 
data expectations, metadata intents, and statements produced resulted in performance measurement gaps 
from some learning activities. In most instances, enough data was generated by other activities to minimize 
the impact of the missing data. 


so The tree view did not utilize vertical screen real estate, staying confined to roughly the top third of the 
page. This caused competencies with large numbers of activities to produce dense packs of associated 
activities, making it difficult for learners to see what was being displayed. 
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4.4.6 Data Analytics and Visualization 


Analytics is the discovery, interpretation, and communication of meaningful patterns in data. It relies on 
the simultaneous application of statistics, computer programming and operations research to quantify 
performance. A variety of visualization techniques are used to communicate insights into the data being 
collected. 


Native LRS dashboards were used to support real time visualization of the Activity Streams as they entered 
into each LRS (system log and participant actions). These dashboards allowed non-technical users to 
participate and understand the analytics process by compiling data and visualizing trends and occurrences. 
The dashboards also provided participants an objective view of performance metrics and served as an 
effective foundation for further dialogue. The 2018 TLA 2018 Reference Implementation exposed 
numerous limitations with the point-to-point messaging that took place between TLA components. This 
approach specifically presented challenges in providing real time dashboards because the data was being 
passed between different localized data stores which required numerous listener services to collect and 
aggregate the data. 


Actor 


Figure 25. Activity Levels of All Learners. 
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In addition to the LRS dashboards, a hack-a-thon was held that promoted further exploration of the data 
generated throughout the week. Hack-a-thon participants performed an exploratory analysis on usage 
patterns across all learners within the TLA 2018 Test and Demonstration. To this end, attendees created a 
variety of charts to explore patterns in user behavior, new ways to use xAPI data, and methods to refine the 
data collection process. As shown in Figure 25, each bar represents a learner, and the height of the bar 
indicates the number of xAPI statements associated with that account. Besides one outlier at the top end 
and many low-activity accounts at the bottom, the distribution is reasonably smooth. 
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The next step was to analyze usage patterns across competencies. Each competency is shown in Figure 26, 
along with the number of x API statements associated with it. The color of the bar represents the badge that 
the competency was placed under. Users were free to work on any competency at any time, but the chart 
below suggests that many may have chosen to begin with the badge positioned at the top (Combat 
Awareness), even though there was no order between the three separate skill trees. 


Object competency ID Object competency badge 
Awareness 
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Wl Other 
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Figure 26. Statements Generated per Learner. — Color Indicates the Specific Competency Framework (Badge) 
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Figure 27. Statements Generated per Competency — Color Indicates Depth. 


In addition, the same data were examined but with respect to competency depth. Each skill tree in the 
FLUENT UI included five level parent-child hierarchy that reflected the CODIAC competency frameworks 
created for this course. Figure 27 provides a chart that shows that the higher-level nodes generated more 
statements, perhaps because they were more easily accessible to learners. To reach a competency at the 
deepest level, users had to click through several parent nodes, whereas the depth 1 competencies were 
immediately visible upon loading the page. 
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Research Question: Were students engaged during their hours with the system? 


Answer: One key advantage of xAPI over older standards is that learner activity can be tracked in much 
greater detail. In addition to completions, successes, and failures, we can now also record actions such as 
turning a page, indicating a preference, or pausing a video. As shown in Figure 28, this allows for a nearly 
perfect reconstruction of a learner’s experience over any span of time. To illustrate this, a prolific user was 
chosen and all associated xAPI statements plotted on a timeline. For each minute of the four-hour period 
from noon to 4:00 PM on Friday, August 17'", 2018, we can see the count of statements generated for that 
user. This is further broken down by color-coding to indicate the verb ID in each statement. 
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Figure 28. Statements per Minute — This chart shows statements by minute for an individual participant, along with 
a color-coded breakdown of the verbs associated with these statements. 


Research Question: Which assessment questions are the most difficult? 


Answer: Another frequently discussed research topic is gauging the difficulty of each task. For longer 
assessments with a wide variety of possible scores, a simple mean or median calculation may not provide 
enough information. Instead, we can examine the distribution of scores to see how different learners have 
performed. One way to do so is to plot the proportion of all learners who achieved a given score (i.e., a 
percentile). As shown in Figure 29, 50% of learners earned a scaled score of 86% or less. 
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Figure 29. Empirical Distribution Function — For a chosen assessment activity. 


Research Question: Can we visualize the learning pathway of a single learner over time? Can multiple 
student pathways through the learning activities be determined, contrasted, and visualized? 


Answer: As shown in Figure 30 the team generated a preliminary visualization for learner paths. Given 
the short duration of the TLA demonstration, most participants were unable to make significant progress 
toward all three badges, so this learning path comparison focused on pairs of students whose final 
competency states were similar. To calculate this similarity, each user’s final competency state for each 
badge and terminal learning objective (TLO) was extracted into a vector. Then a difference vector was 
constructed for each pair for learners by subtracting one learner’s vector from the others. Finally, the norm 
(length) of each pairwise difference vector was calculated (a smaller norm indicating less distance between 
their final outcomes, hence greater similarity). 
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Figure 30. Competency States and Learner Paths — The top chart compares two participants in terms of their final 
competency states. The bottom charts show the learning paths taken by the two users. The chosen users were the two 
most similar ones in the demonstration. 
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4.4.7 Lessons Learned 


Data quality is of utmost important for analytics. The 2018 TLA Reference Implementation exposed 
numerous issues that inhibited analytics and data visualization, both with the data sets themselves and the 
way the data travel through the TLA. The 2018 demonstration did not reach full compliance with the 
guidelines laid out in the 20/8 TLA Data Requirements document. Compliance with the established 
standards is crucial for data consistency and component interoperability. Data issues included the following: 


e Non-conformant usage of context Activities object. Parent and object were frequently the same 
activity. This object should maintain the relationship between the content and its competency node. 


e Reuse of statement IDs. It is crucial to have a unique identifier for each xAPI statement. Per the 
specification, statements should not be overwritten, but instead voided if they must be invalidated. 


e Lack of functionality for YouTube video pages. For video content hosted on YouTube, there was 
no way to track the user’s platform, how the user was directed to video, or how the video linked to 
the competency framework. This resulted in malformed xAPI statements. 


e Inconsistency in actor, verb, object naming patterns. Many statements were missing fields, such 
as the object activity definition. 


e Non-unique IDs and/or other fields being used as unique identifiers. It is essential to have at 
least one unique identifier for each entity type, and uniqueness must be maintained. 


e Inconsistency in verb language. For example, some statements used “en” for the English 
description of a verb, while others used “en-US.” 


e No established timestamp format. Timestamps were recorded to varying levels of precision. 
Some contained time zone information while others did not. As this can cause significant disparities 
between components, a standard timestamp format should be established. 


e Inconsistent labeling of competency levels. Internal competency data (micro, macro, 
competency) did not match external documentation (ELO, TLO, badge), and varying sources 
describe different relationships between these terms. 


e Poorly formatted objects. Different objects followed different schemes for capitalization, space 
padding between fields, and field order. 


One of the biggest challenges that arose was that FLUENT object IDs used a 24-digit hex-identifier that 
was non-conformant with the object identifiers used within an xAPI°! statement. This inhibited CaSS from 
generating assertions for the xAPI statements and required modifications to the approach for processing 
competency assertions from different activities. This had downstream effects as other systems tried to 
access the nonconformant data (e.g., LRS to LRS communications). There is also concern that many of the 
operations within FLUENT were updated in place (e.g., the Activity Index and metadata fields like 
popularity and emotional rating) which resulted in the loss of valuable time-series data. Had these values 
been incorporated into the xAPI statements, the statements would have served as an immutable record of 
these changes and could have been the basis for iterative feedback loops where it would be possible to 
report on the correlations between content properties, learner preferences, and performance. 


>! an xAPI object ID needs to be a URI 
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Assessment: As an Activity Stream, xAPI has no rubric for rolling xAPI statements up into meaningful 
assessment data. Future reference implementations should investigate how an xAPI profile might map to 
HPML’s Human Performance Data Model as evidence of competency. The CASE™ rubric is also suited 
for establishing the criteria and performance levels for earning Credentials. Both may be relevant to 
different types of competencies. 


Granularity of Data: The granularity of data being collected from different LRPs was inadequate and 
often consisted of data like that being generated from a SCORM course delivered via an LMS. Increased 
granularity will enable more robust analytics that each platform can deliver. The use of an xAPI Profile 
Server will help create conformity through controlled libraries and extensions. 


xAPI Data Storage: A separate LRS was used to store system log data due to performance concerns. The 
second LRS mitigated the risk of high network traffic slowing down a single LRS. The use of two different 
LRSs presented difficulties for real-time analytics because the two data streams must be merged before 
analysis can be performed. When real time dashboards are required, all data used in those dashboards should 
be stored in a single LRS. 


Competency Organization: Instructional content was organized into skill trees. This allowed 
competencies to be placed at any desired level of depth, but each node could only have one parent. This 
prevented the many-to-many relationships that competencies are expected to have in a framework. Moving 
forward, competency frameworks should be structured into acyclic directed graphs to preserve 
directionality and eliminate the single parent constraint. 


Learner Profiles: The Learner Profile was used to house mastery estimates for the competencies each 
participant was obtaining throughout the week. It was closely coupled to CaSS but FLUENT used xAPI 
assertions to write learner inference information to the Learner Profile. This approach proved successful 
and warrants additional exploration as a mechanism for collecting assertions about a learner from 
authoritative sources within the DoD. 


Search: No search service was provided in 2018. Participants desired the ability to find the different pieces 
of information they required to supplement the existing learning materials. Searches could be scripted to 
occur through internal DoD database structures or across all content associated with a TLA instance. 


Content Categorization: The number of required LRMI metadata fields used did not create enough 
differentiation between activities for the recommender. For example, different types of text documents such 
as peer-reviewed journal articles, field manuals, guide books, technical reports, and others were all listed 
as a text document. Schema.org presents a vocabulary that can be utilized to add this specificity to the 2019 
Reference Implementation. 


Weighting: All content was treated the same and incremental increases in competency were assigned for 
each completed activity. Weighting was included in the metadata generation spreadsheet; however, 
weighting was not included in the 2018 TLA Data Requirements document so some systems were not able 
to support this capability. Future implementations will require the weighting of evidence based on the 
metadata. 


Service Separation: Many distinct applications and services were run on the same AWS instance with no 
separation. This resulted in undesirable side effects, such as resource contention. Services should be 
decoupled. 
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Data Types: While many of the data elements were known, there were still many undefined data types 
included within the 2018 TLA requirements document. Data types such as activity metadata, relations of 
competency objects, and elements of the Learner Profile need to be better defined so that dashboard 
requirements can be defined. 


Messaging: Multiple messaging formats including REST and AMQP were used to pull and push messages 
between TLA components. Messages were not optimized or organized which resulted in poor data 
discipline by creating multiple instances of the same message. This had a secondary impact of creating 
more network traffic than was necessary and impeded the ability of the 2018 architecture to scale. 


Service Discovery: A service discovery mechanism used either a REST API or an admin portal to register 
endpoints for each TLA service. The service was under-utilized and many TLA components in 2018 used 
hard-coded IP addresses inside their software. This was problematic when trying to set up a development 
environment that is separate from the live production servers after the production servers were locked down 
prior to the event. For future instances of the TLA, a dynamic Service Discovery Protocol (SDP) will be 
investigated to help reduce the configuration efforts required by TLA developers. 
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5. CONCLUSION AND NEXT STEPS 


The 2018 TLA Test and Demonstration, coupled with the 2018 TLA Reference Implementation, the 
CODIAC content, and the 2018 technical specifications proved successful in demonstrating the potential 
of competency-based learning, the viability of existing technical standards, and the amount of relevant data 
that can be generated through the TLA. The 2018 TLA Test and Demonstration also exposed technical gaps 
in the way the TLA was implemented and revealed numerous lessons learned on the complexity involved 
with migrating a single course from a linear program of instruction into a competency-based learning 
system. One of the biggest challenges faced was the attempt to demonstrate the value of data analytics and 
lifelong learning in the context of a single, one-week course. 


Future events should better represent the complexities of different learning environments, different learning 
activities, and student behavioral changes that occur throughout their career. One big revelation during this 
year’s event was the reliance on the training and education community to generate evidence of competency 
within CaSS. From the perspective of competency-based learning, evidence needs to be collected from 
numerous sources including the operational work environment. The 2018 experiment showed this is 
possible from a technical perspective, but that there are many policy issues that need to be changed to enable 
this capability for the future. 
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Figure 31. 2019 Technical Specifications and Standards — This figure shows the preliminary specifications and 
standards being investigated for inclusion in the 2019 TLA Reference Implementation. These show the breadth of 
scale and the scope of impact in migrating away from simply collecting performance data from a learning activities 
and places additional focus on the collection of evidentiary data from operational systems and mission outcomes that 
include key performance indicators that are aligned to the myriad of different competencies in a framework. 
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As shown in Figure 31, the 2018 TLA Test and Demonstration informed the team on new potential 
standards and specifications that should be explored and evaluated for potential inclusion in the future. 
Many of these technical standards are not the atypical learning standards that the ADL Initiative has 
traditionally worked with. Instead, they tie into other areas of the DoD including Failure Modes and Effects 
analysis (FMEA), business rules and logistics, the JCIDS requirements development process, and even the 
way we acquire learning tools, systems and content. Many of the candidate specifications shown above are 
being investigated with the overall intention of positioning the xAPI as a Global Information Grid (GIG) 
standard to enable it to be used to pull data about learner performance from the same work system being 
used on the job. 


The 2018 TLA Reference Implementation showed that while technically feasible to collect and process data 
derived from any system, the policy issues are a limiting factor in the DoD today. One example is during a 
hack-a-thon leading up to the event where a 911 database from NYC was instrumented with xAPI so all 
calls over a specific time frame would be decomposed, quantified into meaningful data, and streamed to an 
LRS for further review. 


Another area of 2019 TLA Test and Demonstration research should focus on creating vocabularies and 
classifications of competencies that can be attached as metadata to each competency object. This would 
allow for commonality and reuse of competency objects and descriptors across organizations. It would 
improve the ability to build assessment strategies that predict the influence of different activity types applied 
to specific competency classifications. Metadata vocabularies might include descriptors that inform 
whether a competency includes psycho-motor skills, fine motor skills, cognitive skills, skill decay, or 
relevant environmental factors that impact or inform the description of a competency. Some competencies 
are linked to the environment in which the competency is expressed, and others are motivated by objectives 
(knowledge, skills, abilities). Table 11 below shows one approach for creating competency classifiers that 
demonstrate how different competencies might be grouped together 


The 2019 TLA Test and Demonstration research should also investigate approaches for formalizing how 
assessment rubrics can be defined, managed, consolidated and shared. Many competency frameworks 
include rubrics, performance levels, or other data that can be used to evaluate proficiency. The Human 
Performance Data Model provides similar capabilities. A TLA Assessment service using common 
vocabularies could provide a common approach for integrating these rubrics into the TLA. 


Table 11. Competency Classifications — This table shows a small number of competency classifications pulled from 
Wikipedia at https://en. wikipedia.org/wiki/Competence_(human_resources). 


Classification Description 


ones These competencies describe the mission, vision, values, culture, and core competency of the 

Organizational ae i : heen: : 

Competency organization that sets the sone and/or context in which the work of the organization is carried 
out (e.g., customer-driven, risk taking, cutting edge). 

_— Capabilities and/or technical expertise unique to an organization. These competencies 

competency differentiate an organization from their competition (e.g., the technologies, methodologies, 
strategies, or processes or that organization that create competitive advantage). An 
organizational core competency is an organization’s strategic strength. 

Technical Depending on the position, both technical and performance capabilities should be weighed 

Competency carefully as employment decisions are made. For example, organizations that tend to hire or 
promote solely based on technical skills, (i.e. to the exclusion of other competencies) may 
experience an increase in performance-related issues (e.g. systems software designs versus 
relationship management skills). 
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Classification Description 


: Individual performance competency is more specific than organizational competencies and 
Behavioral les : : . ’ : 
capabilities. As such, these competencies are defined in a measurable behavioral context in 
Competency ‘ aly : ; 
order to validate applicability to the work environment and the degree of expertise. 
Functional Functional competency is job-specific competency that drives quality results for a given 
position. They are often technical or operational in nature (e.g., backing up a database). 
Competency 
ijanapanant Management competency identifies the specific attributes and capabilities that illustrate an 
8 individual’s management potential. Unlike leadership characteristics, management 
Competency aoe : an 
characteristics can be learned and developed with the proper training and resources. 


To mitigate the limitations of the point to point architecture used in the 2018 TLA Reference 
Implementation, a real-time stream-processing software platform is needed to handle the numerous data 
feeds in the TLA. As shown in Figure 32, Apache’s open-source Kafka platform is one such solution. This 
migration will provide consistency in how data is accessed and communicated throughout the TLA. TLA 
components will subscribe to relevant TLA data streams (topics), pull from various data stores within the 
TLA data lake, and publish new TLA data streams that other components will subscribe to. In the future, 
this exploration may identify new capabilities and potential for managing data about learning. 
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Figure 32. 2019 Data Streaming Architecture — The 2019 TLA Reference Implementation is migrating into a more 
modern data streaming architecture using Apache Kafka. Data producers are along the top and may include learning 
activities, biometric feedback devices, mastery estimates, Learner Profile updates, or any other types of data that need 
to be collected in support of education and training. 


71 The ADL Initiative — May FY19 


Using the streaming architecture, we will be able to stream numerous types of data in any number of 
formalized data schemas that represent the numerous types of data that a typical learning ecosystem might 
generate. The concept is to forward all data to a brokerage service that validates the messaging format and 
sends the appropriate data topics to the appropriate endpoint in the data lake. This provides a single 
collection point that all other TLA components can publish or subscribe to in real time while also providing 
the ability to pull in historical data from the data lake. 


The vast increase in the quantity of personal information that is being collected and retained, combined 
with the increased ability to analyze it and combine it with other information, is creating valid concerns 
about the ability of different TLA components to manage these unprecedented volumes of data responsibly. 
There is an urgent need to strengthen the underlying systems, component products, and services that make 
learning data meaningful. Data labeling will enable the implementation of artificial intelligence, 
recommendation engines, and adaptive tutors by creating a uniform approach for labeling data within the 
future learning ecosystem. In addition to a data labeling strategy, an exploration into simulated datasets will 
present the opportunity to start exploring other relevant topics such as the data lifecycle, periodicity of 
required updates, and pattern discovery techniques. 


As previously mentioned, the 2019 TLA Reference Implementation will also instantiate the Learner Profile 
as separate entity from CaSS. The promise of new TLA applications stems from the ability to create, collect, 
transmit, process, and archive information on a massive scale. The vast increase in the quantity of personal 
information that is being collected and retained, combined with the increased ability to analyze it and 
combine it with other information, is creating valid concerns about the ability of different TLA components 
to manage these quantities of data responsibly. This work will identify best practices for managing a Learner 
Profile across the continuum of lifelong learning while also defining approaches, processes and 
methodologies for creating, managing and storing Learner Profile data and describing lifecycle 
considerations for each data type including authoritative sources where each data type can be discovered. 


An Activity Registry is an approach to capturing, connecting and sharing data about learning resources 
available to an organization. Key features include the ability to generate and manage content metadata, 
manage taxonomies and ontologies, alignment of content with competencies, generate and manage paradata, 
perform semantic search services, and create machine-actionable metadata for Al-based recommenders. 
Organizations that consume a learning resources will also capture information about how a resource is used 
including context, user feedback, user ranking, rating, annotations, etc. Over time, this paradata (usage data) 
and third-party analytical data may become more valuable and more useful than curated cataloging 
metadata for discovery and understanding which learning resources are effective. 


A big effort as part of the 2019 TLA Reference Implementation work includes enhancing the metadata 
strategy using the LRMI specification to build out required, recommended, and optional metadata attributes 
for each learning activity. Additional research needs to identify an approach for describing various learning 
activities a learner will encounter across the continuum of lifelong learning. Best practices will also be 
identified for managing, updating, and versioning this information in an intuitive way that can also drive 
the development of a common course catalog across the entire DoD. The Data Analytics and Visualization 
Environment (DAVE) Framework provides an open-source means of creating domain-based learning data 
dashboards that is extensible to new learning problems, instructional strategies, technological realities, and 
metrics objectives. As shown in Figure 33, each topic in the DAVE Framework features a domain problem, 
an algorithm designed to solve the problem, and a recommended data visualization. 
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Figure 33. 2019 Data Analytics and Visualization Environment — The migration to a Data Streaming architecture 
in 2019 will enable a common approach for accessing TLA data through micro-services that publish and subscribe to 
different TLA data streams. This approach allows TLA dashboards to pull data from the different streams and 
transform that data into meaningful reports, visualizations, and analytics. 

The migration to the new architecture also provides the ability to subscribe to data from any TLA 
component to produce specific data-driven visualizations that represent data and/or analytic results in 
concise, graphical ways. These analytics are intended to go beyond static charts and graphs to produce real- 
time reporting capabilities. While these reports can be used to provide an answer to a specific question, 
they should also promote further inquiry and exploration of the vast amount of data collected about different 
elements of training and education. Within the TLA, a data dashboard is expected to be an information 
management tool that visually tracks, analyzes, and displays key performance indicators (KPI), metrics, 
and key data points about an individual, class, a training event, or other TLA-related information. This 
requires data to be aggregated from across TLA components. TLA dashboards will combine data from an 
LRS with learner data, content paradata and metadata, competency frameworks and other relevant data to 
provide insights and reports. 
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